Futuremark Announces Patch for 3DMark03

Is 52.16 the only affected driver? ( Not 52.14, 52.13? )

It affects all drivers (any version past 44.03)... NVidia reimplemented these optimizations in 44.67 (but if you want an official NVidia version, try 45.23)...

It will affect all of the Det 50s to date (up to 52.16, including all prior versions)
 
I'm Sorry but how can Futuremark approve drivers that we can see with our own freakin eyes are 'cheating' according to Futuremark optimization guidelines !!

If they were not cheating then the results after the patch would be the same, but obviousally NOT.

This fiasco gets more idiotic every day, so everytime there is a new Forceware Driver release, count on a Futuremark patch :LOL:
 
Joe DeFuria said:
Yes, that seems to be the case. FM highlights that these drivers are aproved only with the 440 patch. Unfortunately, I don't like this approach.

I would much rather that FM simply not approve any drivers when a "patch" is required to force the driver to follow the guidelines. They should still release the patch though...so that the drivers give the "correct score" to the best of FM's knowledge.

The way it seems to be now, there is little incentive for the vendor not to cheat! Now, all the burden seems to be on FM to "circumvent" the cheats in order to have a set of "approved drivers" to test. Significant burden should be on the IHV.

Seems very much backwards to me. :cry:

From the looks of it FM is simply starting everyone off with a clean slate. Any cheats found in the most current drivers have been blocked, and everyone's most recent drivers get "approved". It helps avoid accusations of playing favorites. I understand the desire to have some punitive consequences for IHV's that try to cheat, but in the end the whole point of this is to ensure accurate reporting of scores. All major IHV's have a current driver release that meets FM's requirements. Dwelling on past actions does little good. Focusing on what the IHV's do from here on out is much more important.
 
Joe DeFuria said:
Don't want to sound too harsh on FM.

The fact that they are releasing patches to circumvent cheats is great.

I can now very much recommend 3DMark03 to be used in benchmark tests...as long as "approved" drivers are used with the patch. FM is to be commended STRONGLY for taking an active stand against cheating in benchmarks.

:D

It's just that to satisfy me almost 100%, any driver that is "caught" cheating, should not be labeled as approved at all IMO.
I'm with you, but I STILL think public floggings of the offending IHV's CEO should be part of it too. ;)

Thanks Worm for the update and thanks Dave for pointing out yet again nVidia's cheating ways. After all their happy-talk recently I was almost getting ready to get my hopes up that they were seriously going to change their ways, it's nice to know that some things are predictable.... :rolleyes:
 
If they were not cheating Tommy, scoring before the patch and after would be the same. The patch overides whatever Nvidia was detecting, and surprise, surprise GT4 or the Pixel Shader 2.0 test was the number one target.

GT4:

37 FPS Patch 330
26 FPS Patch 340
 
Deathlike2 said:
Joe... it's 340 :)

Thanks!

I guess that can be the case, but one of two things should happen then.

1) Futuremark should let everyone be aware of this fact. If these drivers are approved because [Patch 340] stops "unauthorized optimizations" in this set of drivers, then that's fine.

2) Futuremark should remove/revoke their approval to this set of drivers.

Yes, I agree with you.

Bottom line is, FM should not label a set of drivers as "approved" if they requrie a patch to defeat cheating.

I do understand FM's position though: that is, FM wants all the hardware running it's benchmark. If you don't "approve" of the drivers, you don't get to test them, which diminishes the value of the benchmark.

We're asking FM to make it clear that it is in fact valid to test the latest dets with the 440 patch, but at the same time, the latest dets are invalid. To be fair to FM, that's not easy to communicate clearly.

I do think an attempt should be made though, because otherwise, (as I stated before), there really is little incentive for nVidia to stop. FM will end up being forced to release a new 3D Mark patch with every WHQL Detonator release.... (Which isn't all that often, but still...) This forces FM to "find the cheats", rather than pressure nVidia to just stop cheating.
 
AzBat said:
I'm lost. Where's the cheating by NVIDIA? Serious question.

Tommy McClain

Code:
                     Patch  Driver    3DMarks   GT1    GT2   GT3   GT4 
GeForce FX 5950 Ultra 330   52.16       6412    205.7  46.6  37    37.3 
GeForce FX 5950 Ultra 333   52.16       5538    205    39.6  33.1  26.3 
Radeon 9800 XT        330   3.8         6435    209.7  45.4  38.6  36.3 
Radeon 9800 XT        333   3.8         6436    210.5  45.4  38.6  36.3
 
DT and Dig,

Thanks for pointing that out. I didn't see that at first. I must still be a sleep this morning. :D

It does question why NVIDIA's score went down. And it does make you wonder why Futuremark approved those same drivers. I'm not going to yell fowl just yet since I think Futuremark owes an explanation of why there's a difference. Maybe there is good explanation for this. Don't get me wrong, I'm not sticking up NVIDIA by no means. I'm just being a little cautious with the damnations. Let's see if Futuremark explains this.

Tommy McClain
 
Correct me if I'm wrong, but did the scores drop more on the 5900 last time FM did a patch for 3DMark, or by the same amount as these? I'm fairly sure the drops were more substantial last time (which is actually a good thing, btw. It means at least some of the "optimisations" are gone). I thought that GT4, in particular, was more harshly affected than with this patch + Det combination.

Could be wrong, of course. My memory isn't what it used to be ;)
 
PaulS said:
Could be wrong, of course. My memory isn't what it used to be ;)

Yes the drob was bigger last time. But they needed to cheat more to get on par with Ati (the NV30 is much worse then the NV35 in pixelshader performance).
 
I believe (don't recall exactly), that both GT4 and the synthetic Pixel Shader tests scores dropped about in half with previous "patch to catch cheating." This time, GT4 "only" (cough) dropped 30%. However, the "old drops" IIRC were with NV30, not NV35. So it's hard to tell if some of the optimizations have been dropped, or that NV35 hardware tweaks have some positive effect.

Anyone test the synthetic pixel shader tests pre and post 340 patch?
 
The press release does state the following...

Futuremark press release said:
Parts of the program code have been changed so that possible 3DMark03 specific optimizations in current drivers will not work. 3DMark03 specific optimizations in drivers are against run rules of 3DMark03, because they invalidate the performance measurement results and thus make it impossible to compare performances of different hardware. A list of drivers that have been tested - and confirmed to produce valid 3DMark03 scores - has been published on Futuremark’s website.

I can understand doing this, but if they do do this and scores are lower than previous builds then they need to explain why. Especially if they're going to appove the drivers. I'm going to go back and read their rules, guidelines and enforcement announcements to see if they're doing what they said they would do. I have a feeling they messed up though.

Tommy McClain
 
AzBat said:
I'm going to go back and read their rules, guidelines and enforcement announcements to see if they're doing what they said they would do. I have a feeling they messed up though.

I certainly don't recall reading that they could / would approve drivers that have been found to not meet their criteria. Othierwise, I wouldn't be surprised that the drivers are certified. ;)

Then again, they probably never expicitly stated what happens to drivers (other than feedback to the beta testers and to the IHV) that are found to be cheating.

I'll go back and have a read myself.

http://www.beyond3d.com/forum/viewtopic.php?t=8082

and

http://www.beyond3d.com/forum/viewtopic.php?t=8767
 
I dunno, I think one of the reasons they approved the 52.16 set WITH the latest patch is to highlight the fact that nVidia is still cheating same as they always have.

A public shaming of sorts, of which I approve. :) (I don't like to scare the millionares, I just like to piss 'em off! ;) )
 
Let's first look at the September 23 announcement. It includes the original rules...

http://www.futuremark.com/pressroom/pressreleases/?092303

Futuremark press release said:
1. It is prohibited to change the rendering quality level that is requested by 3DMark.

2. It is prohibited to detect 3DMark directly or indirectly. In its sole discretion, Futuremark may approve detection in order to fix a specified hardware error.

3. Optimizations that utilize the empirical data of 3DMark are prohibited.

4. Generic optimizations that do not violate the above rules and benefit applications in general are acceptable only if the rendering is mathematically consistent with that of Microsoft® DirectX® reference rasterizer.

As a summary, all 3DMark specific optimizations are prohibited. Additionally, all generic optimizations that change the rendering quality requested by 3DMark are prohibited.

First thing we got to ask ourselves is why did the scores go down? Was NVIDIA detecting the application? If so, was this for cheating or was this to "fix a specified hardware error"? You would think that if it was cheating that Futuremark wouldn't approve the driver. So maybe Futuremark approved the detection because it fixed a hardware error? If so, then why didn't explain this? If NVIDIA wasn't detecting the application, then why would their score drop? Could it be due to the maximum point sprite size change or something else they changed? Lots of questions need answering.

I'll continue with other official announcements in subsequent messages.

Tommy McClain
 
Theres something fishy with this new patch. With this new patch my 9700np @303/303 can now beat an fx 5900 nu and fx 5950 ultra in mother nature and the vertex shader test but even against the fx 5900 nu it loses in the pixel shader 2.0 test :?

That doesnt seem to make sense to me if they really have disabled all app specific optimizations (the one test it definitly should win in i.e. full precision ps2.0)
 
digitalwanderer said:
I dunno, I think one of the reasons they approved the 52.16 set WITH the latest patch is to highlight the fact that nVidia is still cheating same as they always have.

A public shaming of sorts, of which I approve. :) (I don't like to scare the millionares, I just like to piss 'em off! ;) )

The answer seems to be a lot simpler and less political than you might think, Dig. :) FM's press release is clear that the only "valid" version of 3DMark2003 is build 340. Under build 340, the 52.16 Dets work the way they're supposed to. Ergo, they're "approved". How they behaved under previous builds of 3DMark isn't relavent to FM's current policies. This was the safest play for FM in order to rebuild their reputation. Wipe the slate clean, start everyone off with valid drivers that don't subvert the current build of 3DMark2003, and let the IHVs' behavior from this point forward speak for itself. If the next set of Dets/ForceWare/whatever brings back the dirty tricks, everyone will know Nvidia's still scum. If they play fair, then maybe Nvidia's learned something.

Personally, I'm more curious how website reviewers will handle things from here on out. FM has set up pretty clear guidelines for how it wants its software used. Will website reviews stick to "approved" drivers only? Or will they use special "preview" drivers from IHV's that may or may not be hacked to high heaven in order to boost scores?
 
has there been investigation as to how those gains were being procured ?

or is that a work in progress dave ?

:)
 
A few things from the "Guideline Enforcement PDF:"

http://www.futuremark.com/companyinfo/Enforcement_Process.pdf

1) There is nothing in their proces map that indicates FutureMark will create patches to deal with "reviewed" drivers. In fact, the driver is only to be identified on 3DMark's site as reviewed if there are "no issues" found.

2) The PDF states: "The objective is to reach a state within the industry, where all new drivers fulfil the guidelines for 3DMark. It is in everyone’s interest that hardware reviewers, end-users of Futuremark’s products and OEMs know that they can trust the benchmark results they obtain with new drivers."

The above seem to go against issuing patches that defeat detection, and at the same time labeling cheat drivers as "reviewed."

However, the PDF also states this:

"The next step in enforcing the guidelines will be a release of a new build of 3DMark03. There will be both full version, and a patch. The patched version with drivers that are listed on Futuremark’s web site as ‘Reviewed’, will produce a valid 3DMark result."

What 3D Mark did with this patch, is consistent with their "next step." The patch, and the drivers listed as reviewed, do produce valid 3DMark scores.

So, I'm left with this line of thought:

Path 340 and the current listed drivers are the BASELINE for 3DMark.

FutureMark informed nVidia of all the invalid cheats it found in nVidia's drivers. FutureMark defeated them. nVidia now has zero excuse to STILL have such cheats in their next WHQL drivers.

If the next drivers out of nVidia still have cheats detected by FutureMark, 3DMark will should NOT label them as "acceptable." 3DMark should still issue a new patch to defeat new cheating mechanisms in the drivers that defeat patch 340 but they should still only approve the Det 52.16 drivers as valid for 3DMark..

So, nVidia has ONE WHQL DRIVER RELEASE to get their act together.

In the end, this is acceptable to me. FM needs to have as much hardware supporting it's benchmark as possible, and by using path 340 (even if it's with drivers that are attempting to cheat), allows this to happen. At the same time, it gives nVidia no excuse for not taking these optimizations out, since 3D Mark obviously identified them and communicated them to nVidia.

Assuming that FutureMark does not label any FUTURE nVidia drivers that attempt to cheat as "accptable for benchmarking", I'm perfectly happy with what they did.
 
Back
Top