New 3DMark03 Patch 330

[Nick[FM]]

FYI, we just released a new patch for 3DMark03.

Check out more info here -> http://www.futuremark.com/news/?newsarticle=200305/2003052308#200305/2003052308

Download -> http://www.futuremark.com/download/?3dmark03patch330.shtml

Audit Report (PDF) -> http://www.futuremark.com/companyinfo/3dmark03_audit_report.pdf

There you go!

 

[Harlequin]

3DMark®03 version 3.3.0
3DMark03 v3.3.0 is an updated version of 3DMark03. Hardware review sites discovered deliberate cheats in some drivers. These drivers identify 3DMark03 build 3.2.0, and render the tests differently than 3DMark03 instructs, in order to gain additional performance. Build 3.3.0 has been changed so that the test results remain the same, but the questionable drivers do not identify 3DMark03 anymore. The drivers now think 3DMark is a 3D application among others, and render the tests like 3DMark instructs. This produces a result that is genuinely comparable to other hardware.

Please read more about this cheating issue on the Futuremark web site (www.futuremark.com).


from the read me.

 

[Joe DeFuria]

Looks like this is going to be a busy thread....just got through the first paragraph of the "audit".

 

[Geo]

Yowza. 24.1% and totally, completely, unmistakeably busted. One assumes that Futuremark had lawyers review/approve that press release.

 

[Patric Ojala]

Yowza. 24.1% and totally, completely, unmistakeably busted. One assumes that Futuremark had lawyers review/approve that press release.

Yes we had.

 

[martrox]

Thank you, Futuremark.....

 

[Clashman]

What Is the Performance Difference Due to These Cheats?

A test system with GeForceFX 5900 Ultra and the 44.03 drivers gets 5806 3DMarks with 3DMark03 build 320. The new build 330 of 3DMark03 in which 44.03 drivers cannot identify 3DMark03 or the tests in that build gets 4679 3DMarks – a 24.1% drop.

Our investigations reveal that some drivers from ATI also produce a slightly lower total score on this new build of 3DMark03. The drop in performance on the same test system with a Radeon 9800 Pro using the Catalyst 3.4 drivers is 1.9%. This performance drop is almost entirely due to 8.2% difference in the game test 4 result, which means that the test was also detected and somehow altered by the ATI drivers. We are currently investigating this further.

Looks like ATI may have had something in there as well, (although not even beginning to approach nVidia in scale).

 

[Joe DeFuria]

Well done, Futuremark.

The evidence was well laid out and detailed. And when you didn't know how something was done (just what the results were)...you said as much.

I also appreciate the "FAQ" approach to the document, and addressing the "non technical" aspects behind this. (Why cheating is bad, what it means to your customers to have as fair a benchmark as possible, etc.) You expressly addressed the key questions that have been raised by those willing to give nVidia the benefit of the doubt. Question like "Is this just revenge..." "Why does this matter...", etc.

Of course, that probably won't stop the same group of people from now saying things like "It's just revenge!" and "It doesn't matter". But there is no denying the valid resons behind your actions.

Looks like you're going to have to more or less be eternally vigilant with respect to keeping tabs on the cheating. You're a victim of your own success. I'm happy to see that so far, you are taking up the challenge, and not ignoring it.

So the question is, how will nVidia respond (or even IF they will respond.). Now, all that driver work for "optimizaiton" went for naught. Will there be another PR or web-site media blitz about how FutureMark is wrong to protect the fairness of their benchmark? Will nVidia "recommed" that sites that choose to use 3DMark "not use" the patch?

Will there be a cycle of "nVidia patches drivers with new detect routines, FutureMark patches 3Dmark with new anti-detection routines updates?"

 

[Mariner]

It's a pity to see that ATI have been caught 'optimising' a bit as well. Obviously their driver team must be rubbish, if they can't compete with the NVidia boys!

It will be interesting to see Kyle's response to this. Outright denial? More mud-slinging? If he comes out and admits he was wrong originally, I'd have much more respect for the fellah.

Let the (cheating) debate begin...

(Again)

 

[galperi1]

oh god...

I just read on 2 forums and guess what....

now that the report is out, everyone is switching from Nvidia didn't cheat because you can't see it to Ati is cheating too.

Now they have yet another mindless defense to spew in favor of Nvidia.

 

[Joe DeFuria]

Looks like ATI may have had something in there as well, (although not even beginning to approach nVidia in scale).

Yup. I would like to see an explanation from both ATI and nVidia about "why" they are detecting.

To be clear....I do NOT believe that "application detecting" in and of itself is a bad thing. There can be "legitimate" optimizations that are done on an appliction by application basis, that hinge on data usage patterns typical of that application. (One app is texture intensive, one is triangle throughput intensive, one is shader intensive....)

It's what the driver does when the app is detected that is or is not a cheat. Anything that takes advantage of a "camera path on a rail" for example, is cheating. Replacing a shader with a "different but similar one", is cheating. Lowering precision is cheating. But I can imagine there are certain things that wouldn't be cheating. ATI's optimization may or may not be a cheat. Likewise, I could see SOME of nVidia's optimizations may or may not being what I consider cheats.

In other words, completely eliminating as much application detection as you can (as seems to be what FutureMark is doing) might eliminate "valid" optimisations along with cheats. That is a very acceptable trade-off though, because "cheats" tend to be much larger in terms of impact.

 

[Joe DeFuria]

What I want to know is...

Did Futuremark address the most obvious cheat....change the name of the executable, "on behalf of Trident" as well?

 

[Reverend]

Personal opinion follows.

For a benchmark suite like 3DMarkXX that has a very definite purpose (and one that is different from games), I think there needs to be a clear statement by FutureMark like "Don't touch my code!" -- provided this is what FM wants because of various possible reasons (like really levelling the playing field, for example).

For games, however, I think the situation may not be so clear cut.

As for the audit report, I thought it was very well done but it is not what I expected to see ( to Aki). Ah well, sometimes lawyers do come in handy I guess.

 

[mondoterrifico]

What Is the Performance Difference Due to These Cheats?

A test system with GeForceFX 5900 Ultra and the 44.03 drivers gets 5806 3DMarks with 3DMark03 build 320. The new build 330 of 3DMark03 in which 44.03 drivers cannot identify 3DMark03 or the tests in that build gets 4679 3DMarks – a 24.1% drop.

Our investigations reveal that some drivers from ATI also produce a slightly lower total score on this new build of 3DMark03. The drop in performance on the same test system with a Radeon 9800 Pro using the Catalyst 3.4 drivers is 1.9%. This performance drop is almost entirely due to 8.2% difference in the game test 4 result, which means that the test was also detected and somehow altered by the ATI drivers. We are currently investigating this further.

Looks like ATI may have had something in there as well, (although not even beginning to approach nVidia in scale).

Yes, but the pdf document clearly states that overall ATI's performance decrease falls within the 3% margin of error.

 

[Moose]

I think this is a GREAT move by Futuremark.

They FOR SURE have caught Nvidia cheating and they may have caught ATI as well and will be investigating further.

This is good for the entire computer industry as well as all computer enthusiasts.

We need to have valid benchmarks so we know how good the products we are going to buy REALLY are.

I think Futuremark has done a good service to help legitimize the benchmarking process.

Kudos.

Although, I suspect that their work in this area will be need to be ongoing to prevent future cheating.

 

[Nite_Hawk]

It will be interesting to see ATI's response to the 8.2% decrease in test4. I hope they come out with a statement one way or another explaining it. Is futuremark issueing any updates on their forums or anything? I'd like to see if they figure out the cause of the decrease. If they could provide a good explaination I could probably buy it's a bug given the marginal decrease in overall performance in 3DMark03. Still, it's rather worrysome.

As for nvidia, I guess I can't say I'm all that suprised. It's sad. I never have really liked nvidia from a moral standpoint, and it had influenced my buying decisions, but not as much as this will. At this point, I don't really feel like I can trust any benchmark I see run on their cards. Even ones from places like Beyond3d. I'm sure Dave would not knowingly publish false benchmarks, but it's impossible to always find everything. Reviewers and futuremark shouldn't be required to babysit the chip/board makers.

Nite_Hawk

 

[Reverend]

Here are some GeForceFX 5200 Ultra scores :

AMD AthlonXP 2000+
512MB
DX9.0a
WinXP Pro

GeForceFX 5200 Ultra -- DetonatorFX 44.03

3DMark03 build 320

3DMark Score		1822 3DMarks
GT1 - Wings of Fury		91.7 fps
GT2 - Battle of Proxycon	10.5 fps
GT3 - Troll's Lair		8.2 fps
GT4 - Mother Nature		9.9 fps
Fill Rate (Single-Texturing)	940.7 MTexels/s
Fill Rate (Multi-Texturing)	920.1 MTexels/s
Vertex Shader		5.5 fps
Pixel Shader 2.0		9.3 fps
Ragtroll			5.8 fps


3DMark03 build 330

3DMark Score		1526 3DMarks
GT1 - Wings of Fury		75.2 fps
GT2 - Battle of Proxycon	9.1 fps
GT3 - Troll's Lair		7.9 fps
GT4 - Mother Nature		7.0 fps
Fill Rate (Single-Texturing)	940.7 MTexels/s
Fill Rate (Multi-Texturing)	920.2 MTexels/s
Vertex Shader		4.1 fps
Pixel Shader 2.0		5.5 fps
Ragtroll			5.7 fps

 

[Evildeus]

At last, that's good news.

 

[Humus]

Ugh!
It was worse than I thought. 24% is a lot.
Those 1.9% from ATi is interesting too. Somehow since it's only in one test and only affects the score a little one possibly theory (for those who refuse to admit ATi is cheating) could be that they have made a workaround for some kind of bug which also affects performance. Will be interesting to see a follow up report from Futuremark on that one.

 

[Nite_Hawk]

wow, overall score is much lower as expected, but look at the pixel shader 2.0 decrease! It's nearly half of what it was with the old driver. How is the pixel shader 2.0 test performed? Is it part of GT4?

Nite_Hawk