Futuremark Announces Patch for 3DMark03

[Bambers]

Can't quite see the fuss myself, I agree with what people have said about starting with a clean slate.

Now obviously if nvidias cheats are reconfigured to work with 340 in their next drivers then these can't be approved. Nor should 3dmark be patched to counter them, I think the ball is pretty much in the IHVs courts atm.

 

[Bouncing Zabaglione Bros.]

Bottom line is, FM should not label a set of drivers as "approved" if they requrie a patch to defeat cheating.

I do understand FM's position though: that is, FM wants all the hardware running it's benchmark. If you don't "approve" of the drivers, you don't get to test them, which diminishes the value of the benchmark.

We're asking FM to make it clear that it is in fact valid to test the latest dets with the 440 patch, but at the same time, the latest dets are invalid. To be fair to FM, that's not easy to communicate clearly.

I do think an attempt should be made though, because otherwise, (as I stated before), there really is little incentive for nVidia to stop. FM will end up being forced to release a new 3D Mark patch with every WHQL Detonator release.... (Which isn't all that often, but still...) This forces FM to "find the cheats", rather than pressure nVidia to just stop cheating.

The thing is that FM want to penalise the cheating drivers merely by (they hope) defeating all the cheats and lowering the 3DMark score. They are not doing what you, me, and several others want ie, effectively disqualify Nvidia's drivers for attempted cheating.

If you think about it, FM are trying to provide accurate 3DMark scores. They don't care if it's done at the driver or at the application, though this does leave the loophole of cheats that they have missed.

For instance, imagine if Nvidia stopped cheating on 3DMark. Their 3DMark scores would go down, but there is nothing to stop Nvidia continuing to cheat on all other benchmarks. "Honest" 3Dmarks scores don't prove that Nvidia isn't cheating in other tests, and merely show that Nvidia cards are slower at the synthetic 3DMark tests.

It seems obvious that Nvidia are willing and able to cheat, and so should be considered to be cheating elsewhere. There is no way for FM to disprove this, and it is outside their remit. All FM want to do is make sure that their 3DMark 2003 scores are clean of cheats, without going as far as pointing at Nvidia and calling them cheaters - even if that is exactly what the results show.

 

[AzBat]

Now I checked the "Optimization Guidelines Q&A" document released on September 26, but I didn't see much there that would help here. However, the "Process to Enforce 3DMark Optimization Guidelines" document released on October 31 does have something that makes you wonder...

http://www.futuremark.com/companyinfo/Enforcement_Process.pdf

If Futuremark suspects that a driver breaches the guidelines, it will contact the manufacturer immediately and demand an explanation or corrective action within one week.

Since ver 52.16 was released on October 23. Could this be the corrective action? Could possibly explain why the scores don't drop down as much as previous versions. Or maybe they didn't give any corrective action, but instead gave an explanation that Futuremark was satisfied with. Or it's intirely possible that Futuremark didn't demand anything and decided to do what others have suggestion and started with a clean slate. This would suggest that guidelines are only valid for drivers going forward. Man, what a freaking mess. Let's continue...

There are two possible outcomes of the above:

1) Futuremark’s suspicion was proved to be wrong and in fact the driver met the Optimization Guidelines. In this case, Futuremark informs its BDP members of the results and publishes information of the driver in its web pages with status: ‘Reviewed’; or

2) The driver did not fulfill the Optimization Guidelines. In this case, Futuremark informs its BDP members of the results and does not publish information of the driver on its web site.

If reasonable evidence is provided, a driver can still be reviewed again at a later time. Futuremark gathers feedback from users of its products, and will make further inspections if needed.

First of all, Futuremark has listed the driver as "Reviewed and Approved". This means that if Futuremarks suspected the guidelines had been breeched, then outcome #1 must have happened. This also means(according to the flowchart) they have notified all the members of the benchmark program as well.

Further inspection of the flowchart shows that other people's comments might be correct that the enforcement of the guidelines only affects new drivers. The flowchart states "New driver is published" at the beginning. To me it looks like that we may never get an explanation on the current NVIDIA drivers because it was "grand-fathered" in. I don't like that at all, but let's see if Futuremark makes announcement on this.

Tommy McClain

 

[Cutter]

has anyone (Dave?) tested the 3.9 Catalysts? Given ATI's openness in dealing with this issue, I'm not expecting anything earth-shattering, but it would be nice to know how it stacks up with the new patch.

 

[Dave Baumann]

Cat3.9's are idential to cat 3.8's WRT to any 3DMark03 version. There don't appear to be any performance tweaks with the 3.9's.

 

[Mariner]

Let's face it - all that Futuremark can do to ensure accurate scores is disable the optimisations in the current drivers. There is no chance that NVidia will say, "Sorry, forgot about those optimisations. We'll remove them from the next driver". To do so would undermine all the FUD they've been very successfully spreading about 3DMark in recent months.

I expect Futuremark will have to release more patches in future as new official NVidia drivers are released, once again featuring such 'optimisations'. I wonder when the first 'optimisation-free' driver will be released? Shortly after the release of the NV40, perhaps?

 

[Bambers]

from the few FX scores that have gone up on the orb it seems the FXs score in the PS2.0 test is unchanged (and still beating 9800s).

futuremark missed one here? :|

 

[Pete]

Finally!

Thanks for the notification, worm. I understand politics may trump honesty, but it's the end result that counts, and FM seems to finally have the cheating licked.

Now let's see what nV does to regain that lost performance with the Det 57's. (I think it's inevitable they'll do something, and that'll be the real test of FM's new policy.)

 

[volt]

Finally!
Now let's see what nV does to regain that lost performance with the Det 57's. (I think it's inevitable they'll do something, and that'll be the real test of FM's new policy.)

Only one thing comes to my mind: write better cheats.

New drivers on the horizon anyone?

 

[digitalwanderer]

Finally!
Now let's see what nV does to regain that lost performance with the Det 57's. (I think it's inevitable they'll do something, and that'll be the real test of FM's new policy.)

Only one thing comes to my mind: write better cheats.

New drivers on the horizon anyone?

New drivers=new patch, betcha a cookie! 8)

 

[Som]

Excuse me good sirs, but


-We had real world benchmarks in games allowing us to compare ATi and Nvidia performance in games.

-We had real world IQ comparisons at NUMEROUS reputable sites allowing us to compare ATi and Nvidia IQ.

-People could use this information to choose between the video cards and purchase what they wanted.

-No one had any complaints.


So, if Nvidia is running "non-standard" code (like that exists or something) in the background to achieve their results, WHO CARES? We know what we're getting when we buy, regardless of whether or not we truly know how we get it.

This is a pretty pointless thread in my opinion. 3Dmark is a pretty useless utility in my opinion. Wait for DX9 games, real DX9 performance tests using those games, and then make your decision. Sheesh. Who cares if they code their drivers differently?

 

[AzBat]

Finally!
Now let's see what nV does to regain that lost performance with the Det 57's. (I think it's inevitable they'll do something, and that'll be the real test of FM's new policy.)

Only one thing comes to my mind: write better cheats.

New drivers on the horizon anyone?

New drivers=new patch, betcha a cookie! 8)

I bet not. I believe they are probably done patching 3DMark03. If and when NVIDIA releases a new driver it won't be reviewed or approved right away. They will take their time to check it out and if it's found it still contains detections or optimizations for 3DMark03, they will demand an explanation or a corrective action. If they don't then all the guidelines, the QA and the enforcement documents were all done for not.

Still want to bet that cookie?

Tommy McClain

 

[nelg]

More details at NordicHardware. Thanks AnteP !

http://www.nordichardware.se/artiklar/Grafik/2003/3DMark03_340/

 

[digitalwanderer]

Finally!
Now let's see what nV does to regain that lost performance with the Det 57's. (I think it's inevitable they'll do something, and that'll be the real test of FM's new policy.)

Only one thing comes to my mind: write better cheats.

New drivers on the horizon anyone?

New drivers=new patch, betcha a cookie! 8)

I bet not. I believe they are probably done patching 3DMark03. If and when NVIDIA releases a new driver it won't be reviewed or approved right away. They will take their time to check it out and if it's found it still contains detections or optimizations for 3DMark03, they will demand an explanation or a corrective action. If they don't then all the guidelines, the QA and the enforcement documents were all done for not.

Still want to bet that cookie?

Sure! (My sister is the manager of a bakery. )

 

[nggalai]

I would much rather that FM simply not approve any drivers when a "patch" is required to force the driver to follow the guidelines. They should still release the patch though...so that the drivers give the "correct score" to the best of FM's knowledge.

In that case, no NV driver after (and including) 44.03 would be approved by FM, making video card comparisons with 3DMark03 practically impossible and thus invalidates the benchmark's selling point ...

93,
-Sascha.rb

 

[Althornin]

Now I checked the "Optimization Guidelines Q&A" document released on September 26, but I didn't see much there that would help here. However, the "Process to Enforce 3DMark Optimization Guidelines" document released on October 31 does have something that makes you wonder...

http://www.futuremark.com/companyinfo/Enforcement_Process.pdf

If Futuremark suspects that a driver breaches the guidelines, it will contact the manufacturer immediately and demand an explanation or corrective action within one week.

Since ver 52.16 was released on October 23. Could this be the corrective action? Could possibly explain why the scores don't drop down as much as previous versions. Or maybe they didn't give any corrective action, but instead gave an explanation that Futuremark was satisfied with. Or it's intirely possible that Futuremark didn't demand anything and decided to do what others have suggestion and started with a clean slate. This would suggest that guidelines are only valid for drivers going forward. Man, what a freaking mess. Let's continue...

There are two possible outcomes of the above:

1) Futuremark’s suspicion was proved to be wrong and in fact the driver met the Optimization Guidelines. In this case, Futuremark informs its BDP members of the results and publishes information of the driver in its web pages with status: ‘Reviewed’; or

2) The driver did not fulfill the Optimization Guidelines. In this case, Futuremark informs its BDP members of the results and does not publish information of the driver on its web site.

If reasonable evidence is provided, a driver can still be reviewed again at a later time. Futuremark gathers feedback from users of its products, and will make further inspections if needed.

First of all, Futuremark has listed the driver as "Reviewed and Approved". This means that if Futuremarks suspected the guidelines had been breeched, then outcome #1 must have happened. This also means(according to the flowchart) they have notified all the members of the benchmark program as well.

Further inspection of the flowchart shows that other people's comments might be correct that the enforcement of the guidelines only affects new drivers. The flowchart states "New driver is published" at the beginning. To me it looks like that we may never get an explanation on the current NVIDIA drivers because it was "grand-fathered" in. I don't like that at all, but let's see if Futuremark makes announcement on this.

Tommy McClain

Or, you are missing the boat, because the 52.16 series driver DOES NOT CHEAT (has no driver optimizations that break guidelines) in 3dmark03 patch 340.

 

[AzBat]

Joe,

Good post. Much better than mine.

A few things from the "Guideline Enforcement PDF:"

http://www.futuremark.com/companyinfo/Enforcement_Process.pdf

1) There is nothing in their proces map that indicates FutureMark will create patches to deal with "reviewed" drivers. In fact, the driver is only to be identified on 3DMark's site as reviewed if there are "no issues" found.

I agree with this. I suspect Futuremark will probably release 3DMark04 before they release another patch.

2) The PDF states: "The objective is to reach a state within the industry, where all new drivers fulfil the guidelines for 3DMark. It is in everyone’s interest that hardware reviewers, end-users of Futuremark’s products and OEMs know that they can trust the benchmark results they obtain with new drivers."

The above seem to go against issuing patches that defeat detection, and at the same time labeling cheat drivers as "reviewed."

However, the PDF also states this:

"The next step in enforcing the guidelines will be a release of a new build of 3DMark03. There will be both full version, and a patch. The patched version with drivers that are listed on Futuremark’s web site as ‘Reviewed’, will produce a valid 3DMark result."

What 3D Mark did with this patch, is consistent with their "next step." The patch, and the drivers listed as reviewed, do produce valid 3DMark scores.

So, I'm left with this line of thought:

Path 340 and the current listed drivers are the BASELINE for 3DMark.

I'm starting to believe this is the case well. Futuremark are getting what they want: valid 3DMark03 scores without having to call them cheats.

FutureMark informed nVidia of all the invalid cheats it found in nVidia's drivers. FutureMark defeated them. nVidia now has zero excuse to STILL have such cheats in their next WHQL drivers.

If the next drivers out of nVidia still have cheats detected by FutureMark, 3DMark will should NOT label them as "acceptable." 3DMark should still issue a new patch to defeat new cheating mechanisms in the drivers that defeat patch 340 but they should still only approve the Det 52.16 drivers as valid for 3DMark..

I don't agree with this. Why does Futuremark have to do this? They have already stated that they won't accept scores that use drivers that aren't approved. So everything on their site that is used officially will use the older approved drivers and show the lower, correct results. There's no need for Futuremark to have to release a new patch with every release of offending driver. If NVIDIA wants results that show their chips are faster, then they will have to release a non-offending driver.

So, nVidia has ONE WHQL DRIVER RELEASE to get their act together.

Agreed. It will be interesting to see what path they choose.

In the end, this is acceptable to me. FM needs to have as much hardware supporting it's benchmark as possible, and by using path 340 (even if it's with drivers that are attempting to cheat), allows this to happen. At the same time, it gives nVidia no excuse for not taking these optimizations out, since 3D Mark obviously identified them and communicated them to nVidia.

Assuming that FutureMark does not label any FUTURE nVidia drivers that attempt to cheat as "accptable for benchmarking", I'm perfectly happy with what they did.

Agreed. Now that I believe I understand what they have done. It's OK with me as well. Now, if they decide they need to release another patch with another driver release, then this process is NOT going to work and they might as well just throw in the towel.

Tommy McClain

 

[AzBat]

Or, you are missing the boat, because the 52.16 series driver DOES NOT CHEAT (has no driver optimizations that break guidelines) in 3dmark03 patch 340.

Umm, that's possible I guess. I have a hard believing that considering the disparity between the results of 2 different builds. Without any comments from Futuremark I guess there is no way of knowing for sure.

Tommy McClain

 

[PaulS]

Or, you are missing the boat, because the 52.16 series driver DOES NOT CHEAT (has no driver optimizations that break guidelines) in 3dmark03 patch 340.

Umm, that's possible I guess. I have a hard believing that considering the disparity between the results of 2 different builds. Without any comments from Futuremark I guess there is no way of knowing for sure.

No, he's saying that BECAUSE of the new patch, the new drivers aren't cheating specifically in 3DMark, because they can't. Without the patch to stop them, yes they're cheating - but that's irrelevant here.

 

[Deathlike2]

I hope that is the correct statement that Althornin meant....