Futuremark Announcement Delayed

[Joe DeFuria]

...We understand that many people did not believe the performance figures when 3DMark03 was published back in February. Only now we are beginning to see DX9 games emerge and what do you know, their performance readings seem to correlate well with what 3DMark03 showed you already in February .

The problem is, Futuremark stopped overtly defending those February results after nVidia got back into the beta program!

In other words: we are now beginning to see DX9 games emerge...and what do you know...their performance readings don't correlate AT ALL with what 3DMark is currently showing.

The last time Futuremark was seen defending the February (non cheat drivers) results, was with the cheat expose PDF. If you had been defending those scores consitently, you wouldn't be in the "credibility mess" (with end users) that you are in today.

Well, here's to hoping we see a good set of guidelines coming soon...including FM's desire to actively enforce it.

 

[Slides]

Some of you would eat your young.

 

[Joe DeFuria]

Some of you would eat your young.

I couldn't fit 'im in the microwave...

 

[Dave Baumann]

Spit-roast!

 

[WaltC]

]
WaltC,

Well, if you think that the absence of verifiable information should and will cause speculation, I can only say that it's sad. The question is, when will people stop speculating, and just wait for official statements and official publications? That would be much better for all of us. And I don't mean this announcement only.

Worm..Heh...You answered your own question. They'll quit speculating when you get the "official statements" out...

Well, the "wise" quote you posted is true, but is there anything wrong with trying? Of course you can't get everyone to be 100% happy and satisfied with everything you do, but if you don't try, you will never accomplish anything. We think it is utmost important that our users are happy with our products, online and offline media find our tools useful & reliable and that the IHV's see our benchmarks are good and reliable DX benchmarks.

My feeling is that your effort in making a top-flight benchmark should the first priority, and making IHVs happy a distant second. As I said and still believe, if you try to please everyone you may well please no one... Ah, well...I tried...

I know that the public (at least here [smile]) is very demanding and want everything out in the open the very second they want and demand. That is simply impossible to do. I think I speak for many other companies as well. Speculating and spreading baseless rumors only makes it harder.

ps. sorry for hasty & short answers, but I am in a big hurry now!

Worm, it's just that the public here and elsewhere considered that you put everything out into the open with your original audit report of a few months ago. Since then, you've reached some kind of accomodation with nVidia that has yet to be explained as well as your audit report explained things then. When people feel that you've explained these things as well as you laid them out in that audit report, you'll probably see the speculation vastly reduced. Then, it will just be a matter of whether people like what you're doing or not, basically. As opposed to speculation...Heh..

 

[Wolf]

So in other words there will be no changes as I read it. I'm glad 8 months ago I deleted your benchmark from my hard drive. Either your benchmark is for the consumer or the IHV's. There is no middle ground as I see it and I see the direction you are choosing to go without even having to read your news release. Sorry, but your about 7 months too late as far as I'm concerned.

 

[StealthHawk]

On the other hand, I'm now unsure whether FM and Valve and Core are being unfair to both nVidia and its large customer base by ignoring the NV3x's pros and cons and blindly hewing to DX9.

How did you come to that conclusion? I thought Core and especially Valve did NOT ignore NV3x's architecture.

 

[T2k]

LOL, check this: http://ccon.madonion.com/servlet/Index?pageid=/vendors/maximumpc/index

It's a FM product tester... the famous online one... LOL!

Here's my score:

Performance Comparison
This is how the performance of top graphics cards compares to the one in your PC.

PC System Performance

--------------------------------------------------------------------------------

Your Computer 100%


NVIDIA GeForce4 Ti 4400
Compare Individually 82%
Info Buy!

NVIDIA GeForce4 Ti 4200
Compare Individually 68%
Info Buy!

ATI Radeon 9700 PRO
Compare Individually 58%
Info Buy!

ATI Radeon 8500 128MB
Compare Individually 55%
Info Buy!

NVIDIA GeForce3 Ti 500
Compare Individually 52%
Info Buy!

NVIDIA GeForce4 Ti 4600
Compare Individually 46%
Info Buy!

ATI Radeon 9000 PRO
Compare Individually 44%
Info Buy!

Matrox Parhelia
Compare Individually 43%
Info Buy!

NVIDIA GeForce3
Compare Individually 42%
Info Buy!

NVIDIA GeForce3 Ti 200
Compare Individually 37%
Info Buy!

SiS Xabre series
Compare Individually 35%
Info Buy!

NVIDIA GeForce4 MX 440
Compare Individually 33%
Info Buy!

ATI Radeon 7500
Compare Individually 22%
Info Buy!

NVIDIA GeForce4 MX 460
Compare Individually 20%
Info Buy!

STMicro KYRO II
Compare Individually 19%
Info Buy!



Nice, huh? It gets even better if you check my current card in my sig...

One more unusable but funny stuff by FM...

 

[ZoinKs!]

Heh, according to that tester, a 9700pro would speed up my system by 23% while a 4ti 4600 would boost it by 18% :?
The tester is broken.

 

[K.I.L.E.R]

Heh, according to that tester, a 9700pro would speed up my system by 23% while a 4ti 4600 would boost it by 18% :?
The tester is broken.

Broken... or rigged?

 

[dan2097]

How the hell does that thing work :?

I have a 9700np flashed to a 9700 pro bios and clocked to 303/303....

then it tells me upgrading to a 9700 pro would drop performance...

R9700 pro =90% of current

 

[PSarge]

Spit-roast!

You sick puppy!

 

[martrox]

You sick puppy!

Good Carl Hiaason book!

 

[digitalwanderer]

Heh, according to that tester, a 9700pro would speed up my system by 23% while a 4ti 4600 would boost it by 18% :?
The tester is broken.

Broken... or rigged?

And you say I'm bashing FM?

 

[WaltC]

Oh, this is just too funny....Here's my report:

(I'm using a stock-clocked 9800P)

Your Computer 100%


NVIDIA GeForce4 Ti 4400
Compare Individually 171%
Info Buy!

NVIDIA GeForce4 Ti 4200
Compare Individually 140%
Info Buy!

ATI Radeon 9700 PRO
Compare Individually 119%
Info Buy!

ATI Radeon 8500 128MB
Compare Individually 117%
Info Buy!

NVIDIA GeForce3 Ti 500
Compare Individually 109%
Info Buy!

Matrox Parhelia
Compare Individually 91%
Info Buy!

NVIDIA GeForce3
Compare Individually 89%
Info Buy!

ATI Radeon 9000 PRO
Compare Individually 88%
Info Buy!

NVIDIA GeForce4 Ti 4600
Compare Individually 87%
Info Buy!

NVIDIA GeForce3 Ti 200
Compare Individually 79%
Info Buy!

SiS Xabre series
Compare Individually 73%
Info Buy!

NVIDIA GeForce4 MX 440
Compare Individually 71%
Info Buy!

ATI Radeon 7500
Compare Individually 47%
Info Buy!

NVIDIA GeForce4 MX 460
Compare Individually 44%
Info Buy!

STMicro KYRO II
Compare Individually 40%

******************************

OMG, what a load!... The program is so stupid that it either doesn't recognize my hardware, or else it thinks I am so stupid as to believe that a GF4 Ti4400 runs 171% as fast as my R9800P...

Unbelievable...They really need to take this down...it's preposterous...it even shows an 8500 as being faster than me and running at the same speed as a 9700P.

IF I was ATi I would definitely look into this...since ATi is still a "partner" and so forth...

Edit: T2K, I am also running a Barton 2500+ @~3000 + level with an FSB clocked to 190MHz. We have similar systems, but got much different numbers. If this is an example of what FM has been doing, then I...well...I'm speechless...and am left with but one question--Why does ATi belong to this silly outfit? (I ran it again and got the identical numbers.)

 

[BRiT]

WaltC, I'll trade you an ATI 8500 for your 9800Pro! That's a 17% performance improvement for free!

 

[WaltC]

WaltC, I'll trade you an ATI 8500 for your 9800Pro! That's a 17% performance improvement for free!

What a deal!... Are you sure? (I wouldn't want to take advantage of newbie, or anything!) I can only do the swap if you feel right about it....

*chuckle*

What really gets me is learning that my 9800P is only 12% faster than a 9000P....! *What* was I thinking when I bought the 9800P!???

Thanks, FM, for setting me straight.

[Not.]

This has to be one of the worst current examples of corrupt marketing I've ever seen--I thought this kind of thing went out of fashion years ago...

 

[RussSchultz]

Corrupt? Those are mighty strong words.

Perhaps its just making its "decision" based on data that is a little out of date?

Like: 3dmk01 scores--which are likely CPU bound, which are likely to only show a 12% increase or so moving to a 9800 pro from a 9200 pro on a particular system like yours.


Note at the bottom of the chart it gives you a little disclaimer:

Performance Analysis is based on a dynamic database of over 5 million benchmark results obtained through submissions from 3DMark2001SE, Futuremark's leading benchmark software.

 

[WaltC]

Performance Analysis is based on a dynamic database of over 5 million benchmark results obtained through submissions from 3DMark2001SE, Futuremark's leading benchmark software.

OK, well, maybe "corrupt" isn't the applicable word--maybe we should use something like "factually bankrupt" or "inexplicably erroneous" to describe it...

The fact is that the tester doesn't tell us anything about 3d cards whatever, although the data represented indicates the difference between systems is based on 3d-card performance exclusively.

Or it could simply mean that as a 3d-card benchmark 3dMK01SE is utterly worthless, since there is zero probablity that a GF4 Ti4400 could ever be 171% faster than my R9800P. Even in cpu-limited cases, when using FSAA and/or AF, it's not even possible that the 4400 is even close to me in performance. Heck, my GF4 Ti4600 couldn't touch my 9700P last year...so I ought to know.

But even more troubling that that, is despite the "disclaimer" (which I don't actually see as a disclaimer, btw--but just fine print that doesn't mean anything), we have no idea *how* the tester software correlates the data it is presumably based on and cannot determine much of anything from its results except that it is wrong. Horribly wrong. Therefore, I do not think we can dismiss the possibility of deliberate misrepresentation.

Edit: Most likely the tester itself is ~ 1 year old without an update--either that or there are not any nV3x '01SE benchmarks recorded, or else they are all so slow [nV3x] they don't even make the charts---perhaps a 4400 is 300% faster than nv30/35...?.... So, it's not only old and completely out of date, it's completely wrong, and it promotes a completely erroneous concept--that you can use the "tester" to determine how fast your system would run with various 3d cards.

 

[BRiT]

The only way I can fathom those results is if one has AA & AF forced on in the control panel while the performance-tester does not detect that. But if the performance tester is just querying the driver for device-id, then the numbers make no sense.