It was only a matter of time

[Myrmecophagavir]

Back to June there are no 5900/5600 or 9800 on the 2003 hall fame list,when I asked worm,he said the results are based on 2001 benchmark,and 5900/9800 benchmark results is very few.

Just 2 months past things changed,oh,so many people now using DX9 cards?

Yes, that makes sense doesn't it? 2 months is a large proportion of the lifespan so far for the 9800 and others, so the number of people using them would have gone up a lot. In addition people weren't so interested in uploading 3DMark2001 scores to the ORB for those cards, people only care about 3DMark03. There was no 03 hall of fame list in June.

 

[dan2097]

I think it's one in GT3 - The "Troll's Lair" test. It's another not clearing the back-buffer 'optimisation' if I remember correctly. It's more difficult to spot, but it's still there.... Am I right?

I thoguht it was a gt2 clip planes, am I right ?

 

[Doomtrooper]

Strategic BETA Members:

"In our testing, all identified detection mechanisms stopped working when we altered the benchmark code just trivially and without changing any of the actual benchmark workload. With this altered benchmark, NVIDIA’s certain products had a performance drop of as much as 24.1% while competition’s products performance drop stayed within the margin of error of 3%. To our knowledge, all drivers with these detection mechanisms were published only after the launch of 3DMark03. According to industry’s terminology, this type of driver design is defined as ‘driver
cheats’.

Members of Futuremark’s BETA program first noticed how parts of the tests in 3DMark03 were rendered differently on different hardware. When testing NVIDIA hardware on 3DMark03 with socalled developer’s version’s free camera enabled, they noticed how some parts of tests were rendered strangely, and informed Futuremark of their findings. Futuremark investigated further and our findings show that certain NVIDIA drivers seem to detect when 3DMark03 is running and then replace the 3DMark03’s rendering requests with manually implemented alternative rendering operations. These alternative rendering operations reduce the amount of rendering work and thereby increase the obtained benchmark result."

And the NVIDIA perspective...

"Since NVIDIA is not part in the FutureMark beta program (a program which costs of hundreds of thousands of dollars to participate in) we do not get a chance to work with Futuremark on writing the shaders like we would with a real applications developer. We don't know what they did, but it looks like they have intentionally tried to create a scenario that makes our products look bad. This is obvious since our relative performance on games like Unreal Tournament 2003 and Doom 3 shows that the GeForce FX 5900 Ultra is by far the fastest graphics on the market today."

Classic ...and predictable

 

[digitalwanderer]

This is just a leetle too much for me this morning, 'scuse me while I go puke up a bit of bile before continuing.

Anyone ever notice that nVidia PR has a wonderful tendancy to really crap up a nice morning?

 

[Matt]

This is just a leetle too much for me this morning, 'scuse me while I go puke up a bit of bile before continuing.

Anyone ever notice that nVidia PR has a wonderful tendancy to really crap up a nice morning?

If a PR representative can affect your morning, then maybe it's time to visit the beach and get some sun.

 

[digitalwanderer]

This is just a leetle too much for me this morning, 'scuse me while I go puke up a bit of bile before continuing.

Anyone ever notice that nVidia PR has a wonderful tendancy to really crap up a nice morning?

If a PR representative can affect your morning, then maybe it's time to visit the beach and get some sun.

Don't EVEN get me started on the e-mail I found from Mr.Perez in me box this morning, it was the freaking icing on the cake.

Beach is out, it's heavy over-cast and me daughter apparently got up VERY early and had some 'fun' around the house before waking us up....I'm still in the "recovering the damage" phase.

 

[Rookie]

Err,Worm said that they have removed the current Hall of Fame:

"Just a quick note to all of you that we have removed the current Hall of Fame. We are working on a better solution and it will be published as soon as it is ready. Stay tuned... "

hmm,Just wondering what better solutions means?

 

[incurable]

hmm,Just wondering what better solutions means?

More nVidia cards on top.

Did I actually write that?

cu

incurable

 

[Joe DeFuria]

hmm,Just wondering what better solutions means?

More nVidia cards on top.

Did I actually write that?

cu

incurable

You may think it's funny, but there might be some sort of real truth to that.

AFAIK There have been issues with certain chipsets reporting as "the same product" to the Database. In other words, a Radeon 9800 and a Radeon 9800 Pro might both be reporting and grouping themselves as "9800". Same with nVidia cards.

This might be a re-vamp to try and actually separate differnt SKUs of the same chip? So you can distinguish a GeForce 5900 Ultra from a 5900, or a ATI 9500 from a 9500 Pro, etc.

If someone at the top has multiple SKUs that are being lumped together, perhaps they made some noise to rectify the situation.

Seriously, I have no idea if this is the reason for the update, and I alsp don't have any idea who would actually benefit more, ATI or nVidia, from a change....

 

[Slides]

Or maybe Matrox will take the no.1 spot...

 

[Nick[FM]]

AFAIK There have been issues with certain chipsets reporting as "the same product" to the Database. In other words, a Radeon 9800 and a Radeon 9800 Pro might both be reporting and grouping themselves as "9800". Same with nVidia cards.

This might be a re-vamp to try and actually separate differnt SKUs of the same chip? So you can distinguish a GeForce 5900 Ultra from a 5900, or a ATI 9500 from a 9500 Pro, etc.

That was already done in the "old" HoF. We already got the (for example) Radeon 9800 and Radeon 9800 Pro, 9500 series and 9700 series all separated etc.

 

[Patric Ojala]

Err,Worm said that they have removed the current Hall of Fame:

"Just a quick note to all of you that we have removed the current Hall of Fame. We are working on a better solution and it will be published as soon as it is ready. Stay tuned... "

hmm,Just wondering what better solutions means?

I'm not sure I know the whole truth, but at least one reason for maintenance was that the old top list was affected by a large amount of 3DMark results run on unofficial drivers. Not leaked beta drivers, wörm just corrected me , but unofficial drivers anyway. We also have our still ongoing effort to define what driver optimizations could be considered acceptable.

That old Top List was basically just some averages of received benchmark results, not some "Futuremark's official best buy guide" as some people saw it as. We'll see what we can do about this list, and hope the new list will be less ambiguous.

 

[K.I.L.E.R]

I'm looking forward to it Mr Ojala.

Err,Worm said that they have removed the current Hall of Fame:

"Just a quick note to all of you that we have removed the current Hall of Fame. We are working on a better solution and it will be published as soon as it is ready. Stay tuned... "

hmm,Just wondering what better solutions means?

I'm not sure I know the whole truth, but at least one reason for maintenance was that the old top list was affected by a large amount of 3DMark results run on unofficial drivers (leaked beta drivers and similar). We also have our still ongoing effort to define what driver optimizations could be considered acceptable.

That old Top List was basically just some averages of received benchmark results, not some "Futuremark's official best buy guide" as some people saw it as. We'll see what we can do about this list, and hope the new list will be less ambiguous.

 

[PatrickL]

[We also have our still ongoing effort to define what driver optimizations could be considered acceptable.

I think it 's the problem. Could you explain please why FM thinks any optimization is acceptable for a benchmark ?

 

[jb]

I agree. NO OPTIMIZATIONS FOR A BENCHMARK should be allowed.

 

[K.I.L.E.R]

I agree. The only optimisations that should be done are API specific optimisations that are all round.

 

[Nick[FM]]

I agree. NO OPTIMIZATIONS FOR A BENCHMARK should be allowed.

Out of curiosity, what about all games with built in benchmarks?

 

[Joe DeFuria]

Once and for all, can Futuremark make a clear and concise statement or set of statements that defines:

1) What the purpose of 3DMark is supposed to be. For example
a) Representative of a guess on absolute performance with DX8 and DX9 games
b) A "stress test" for GPUs, not designed to be indicative of absolute performance levels, but an indiciation of relative GPU power in given circumstances....and what the circumstances are designed to represent.

2) Qualify exactly what 3DMark "score" represents in your view.
a) Measurement of which GPU is "faster"
b) Measurement of Which GPU is "better", (and what, qualitatively speaking, defines "better")

In my opinion, the real problem with 3Dmark, is what appears to me to be an inconsistent presentation from FutureMark of what exactly 3DMark is, how it's supposed to be used, and what the score represents.

I really have no clear indication at this time WHAT exactly the 3DMark score is trying to convey. And until FutureMark can make a CLEAR and CONCISE statement about what the PURPOSE of the benchmark is, it is near impossible to discuss on whether or not it meets those objectives, or what is and is not "acceptable" in terms of optimizations, etc.

And yes, I've read the initial white paper, and I THOUGHT I had understood the purpose...right up through the denouncing of nVidia cheats...but recent events leave me perplexed.

 

[PatrickL]

I don't think you should move the talk to games used as a benchmark after the UT2003 issue. Or do you suggest that the trilinear filtering was removed in Nidia drivers for general gaming purpose as a general optimization

 

[Doomtrooper]

]

I agree. NO OPTIMIZATIONS FOR A BENCHMARK should be allowed.

Out of curiosity, what about all games with built in benchmarks?

Depends on what the 'optimization' is, lets take the ever popular UT 2003 benchmark which has obviousally got some 'special attention' latley. Disabling trilinear is not a 'optimization' as you can see the image quality loss.

http://www.beyond3d.com/forum/viewtopic.php?t=7228&start=0

Replacing shaders in a benchmark that does not change the visual output like replacing the water shader would be welcomed in a 'game' if performance was increased and if the output stays the same vs the orginal reference, but Nvidia has failed on all accounts.

FX

http://www.bit-tech.net/images/review/187/gt4fxlq.jpg

9700

http://www.bit-tech.net/images/review/187/gt4radlq.jpg