Futuremark Announces Patch for 3DMark03

[paju]

The way I see it is that FM had to stop cheats from IHV drivers by changinf their code and performing counter actions. nVidia or ATI sure would not start this if they were just asked to. This way FM can show that there is funny things going on and they don't humiliate IHVs but just show the result.

This leads to exactly what Joe wrote: nVidia has time till next WHQL drivers to get rid of the cheats. And it's also FM's responsibility to not approve any of the newer drivers if they require patching of 3DMark03 to get rid of cheating.

Of course this could lead to very good cheats that are not detected easily. We can hope and vote with our wallet that it won't go there.

Som: I used to think 3DMark03 is waste of time when they changed their statement from cheating to optimization. But now FM is pulling their act together and it seems that results from 3DMark03 can be compared again. It has shown already results of nVidia's and ATI's graphics cards that are in line with those few DX9 games.

 

[Hellbinder]

Excuse me good sirs, but


-We had real world benchmarks in games allowing us to compare ATi and Nvidia performance in games.

-We had real world IQ comparisons at NUMEROUS reputable sites allowing us to compare ATi and Nvidia IQ.

-People could use this information to choose between the video cards and purchase what they wanted.

-No one had any complaints.


So, if Nvidia is running "non-standard" code (like that exists or something) in the background to achieve their results, WHO CARES? We know what we're getting when we buy, regardless of whether or not we truly know how we get it.

This is a pretty pointless thread in my opinion. 3Dmark is a pretty useless utility in my opinion. Wait for DX9 games, real DX9 performance tests using those games, and then make your decision. Sheesh. Who cares if they code their drivers differently?

Actually there are some Complants even in "real World Tests".

First post... Interesting timing on your Arrival and the specific opinion path and view of the information

 

[AlphaWolf]

Excuse me good sirs, but


-We had real world benchmarks in games allowing us to compare ATi and Nvidia performance in games.

-We had real world IQ comparisons at NUMEROUS reputable sites allowing us to compare ATi and Nvidia IQ.

-People could use this information to choose between the video cards and purchase what they wanted.

-No one had any complaints.


So, if Nvidia is running "non-standard" code (like that exists or something) in the background to achieve their results, WHO CARES? We know what we're getting when we buy, regardless of whether or not we truly know how we get it.

This is a pretty pointless thread in my opinion. 3Dmark is a pretty useless utility in my opinion. Wait for DX9 games, real DX9 performance tests using those games, and then make your decision. Sheesh. Who cares if they code their drivers differently?

If Nvidia cares enough to cheat, I care to know about it. This has been gone over a million times but OEMS use scores like 3dmark to choose cards.

An individual game test will tell you about 1 game, 3dmark is attempting to show probable results of the hardware in all games. Also game benchmarks aren't 'policed' by the game developers. Futuremark has a vested interest in maintaining the integrity of the results for 3dmark.

So far 3dmark has been a fairly accurate predictor once you weed out the cheats. I see no reason why it can't continue to do so.

 

[AzBat]

I hope that is the correct statement that Althornin meant....

I agree, because it didn't look like that to me.

Tommy McClain

 

[digitalwanderer]

Excuse me good sirs, but


-We had real world benchmarks in games allowing us to compare ATi and Nvidia performance in games.

-We had real world IQ comparisons at NUMEROUS reputable sites allowing us to compare ATi and Nvidia IQ.

-People could use this information to choose between the video cards and purchase what they wanted.

-No one had any complaints.


So, if Nvidia is running "non-standard" code (like that exists or something) in the background to achieve their results, WHO CARES? We know what we're getting when we buy, regardless of whether or not we truly know how we get it.

This is a pretty pointless thread in my opinion. 3Dmark is a pretty useless utility in my opinion. Wait for DX9 games, real DX9 performance tests using those games, and then make your decision. Sheesh. Who cares if they code their drivers differently?

Actually there are some Complants even in "real World Tests".

First post... Interesting timing on your Arrival and the specific opinion path and view of the information

"So, if Nvidia is running "non-standard" code (like that exists or something) in the background to achieve their results, WHO CARES? We know what we're getting when we buy, regardless of whether or not we truly know how we get it."

This is the second time in 10 minutes I've seen this same assignine argument used by a one-post-wonder. I was gonna call you 'paranoid', but...

(The Dig pulls out his trusty tinfoil hat and carefully unfolds it and places it firmly on his head making sure it completely encases his brain.)

 

[Doomtrooper]

There will be another Futuremark patch following the next driver release from Nvidia, so once we figure out how many Nvidia drivers releases are left this year, we can then extrapolate that into how many patches will be released from Futuremark

 

[Joe DeFuria]

So, if Nvidia is running "non-standard" code (like that exists or something) in the background to achieve their results, WHO CARES? We know what we're getting when we buy, regardless of whether or not we truly know how we get it.

I care, and no, we don't know what we're getting if nVidia is using app specific code to prop up benchmarks.

Why?

Because you cannot tell me that nVidia will put app specific code in their driver for every single application out there. At best, nVidia could put app specific code for popular games and for benchmarks.

So what happens when I have a game that has no "special app detection" in the drivers?

1) You end up having to wait for a new nVidia driver release to play the game with expected performance

or

2) You never get drivers updated for that game.

This is a pretty pointless thread in my opinion.

This is one of the most important threads wrt drivers and cheating in quite long time.

Wait for DX9 games, real DX9 performance tests using those games, and then make your decision.

Unfortunately, that doesn't help someone who's buying a new card NOW, does it? I can be buying a card to play my EXISITNG games better...this doesn't mean I shouldn't consider the possibilities with future games.

In other words, if Card A and Card B both play "current games" very well....but "future benchmarks" (like 3DMark, aquamark, Shader Mark, etc.) show that Card As performance and quality is solid and predictable, and Card B is "flaky...depends wildly on drivers and acceptance of certain quality degredations"....

What card would you buy?

 

[Joe DeFuria]

Hi Tommy!

I don't agree with this. Why does Futuremark have to do this? They have already stated that they won't accept scores that use drivers that aren't approved. So everything on their site that is used officially will use the older approved drivers and show the lower, correct results. There's no need for Futuremark to have to release a new patch with every release of offending driver. If NVIDIA wants results that show their chips are faster, then they will have to release a non-offending driver.

I understand what you're saying, but I still think 3DMark will need to release patches....because even though scores won't be officially submitted to the DataBase for non approved drivers, that won't stop reviewers from using "the latest nVidia drivers" and testing them with 3DMark in their reviews. (Against the license agreement.)

It's just a safeguard against improper use of their benchmark.

Agreed. Now that I believe I understand what they have done. It's OK with me as well. Now, if they decide they need to release another patch with another driver release, then this process is NOT going to work and they might as well just throw in the towel.

My only disagreement here, is that they should issue a patch, but the newer drivers should not be approved.

 

[AzBat]

Hi Tommy!

I don't agree with this. Why does Futuremark have to do this? They have already stated that they won't accept scores that use drivers that aren't approved. So everything on their site that is used officially will use the older approved drivers and show the lower, correct results. There's no need for Futuremark to have to release a new patch with every release of offending driver. If NVIDIA wants results that show their chips are faster, then they will have to release a non-offending driver.

I understand what you're saying, but I still think 3DMark will need to release patches....because even though scores won't be officially submitted to the DataBase for non approved drivers, that won't stop reviewers from using "the latest nVidia drivers" and testing them with 3DMark in their reviews. (Against the license agreement.)

It's just a safeguard against improper use of their benchmark.

Agreed. Now that I believe I understand what they have done. It's OK with me as well. Now, if they decide they need to release another patch with another driver release, then this process is NOT going to work and they might as well just throw in the towel.

My only disagreement here, is that they should issue a patch, but the newer drivers should not be approved.

The time that passes between the release of a NVIDIA driver that cheats and when they can release a driver to defeat it could get really long. So there is already going to be improper use of their benchmark during that time. But according to their guidelines NVIDIA will be forbidden to use the cheating drivers with 3DMark03 in any official capacity. Plus, Futuremark themselves will not use the drivers in the Hall of Fame, etc. If web sites start using them in reviews(against the license agreement) I would hope people would notify Futuremark about it and let them handle them.

If we start seeing Futuremark releasing a patch every time a cheating driver is released then nobody will use the benchmark because Futuremark is basically saying all versions are invalidated. Who's going to want to buy and use 3DMark then? Nobody.

Tommy McClain

 

[Althornin]

Or, you are missing the boat, because the 52.16 series driver DOES NOT CHEAT (has no driver optimizations that break guidelines) in 3dmark03 patch 340.

Umm, that's possible I guess. I have a hard believing that considering the disparity between the results of 2 different builds. Without any comments from Futuremark I guess there is no way of knowing for sure.

No, he's saying that BECAUSE of the new patch, the new drivers aren't cheating specifically in 3DMark, because they can't. Without the patch to stop them, yes they're cheating - but that's irrelevant here.

exactly.
i fail to see how my statement could possibly have been taken to mean anything else.
Lets deconstruct it:
"the 52.16 series driver DOES NOT CHEAT in 3dmark03 patch 340".
seems pretty unequivocable. Cheating woudl be performance boosting, and patch 340 breaks the cheats for 330, ergo, the driver CANNOT be cheating in 340. If i sound pissed, i kinda am, because i stated things pretty clearly - if you can come up with another RATIONAL (and based on english usage) meaning, please, show me.

 

[Joe DeFuria]

F*ck you Tommy! How dare you disagree with me!

The time that passes between the release of a NVIDIA driver that cheats and when they can release a driver to defeat it could get really long. So there is already going to be improper use of their benchmark during that time.

Good point.

But according to their guidelines NVIDIA will be forbidden to use the cheating drivers with 3DMark03 in any official capacity. Plus, Futuremark themselves will not use the drivers in the Hall of Fame, etc. If web sites start using them in reviews(against the license agreement) I would hope people would notify Futuremark about it and let them handle them.

Another good point...although this raises the "who will use 3DMark then?" quesiton. In other words, if FM starts threatening web sites with legal action, are web sites going to bother giving FM support?

If we start seeing Futuremark releasing a patch every time a cheating driver is released then nobody will use the benchmark because Futuremark is basically saying all versions are invalidated.

No, that's not true. All results from patch 340 and the "currently approved" drivers will be valid, now and forever. Once (if) a new patch is released, those older results will not go away, and FM can also still accept results from patch 340 with the "patch 340 approved hardware/drivers".

 

[Hellbinder]

I think as long as the OEMs see a benefit to using 3dmark and see that the field is as level as possible it will be ok for FM.

 

[AzBat]

Or, you are missing the boat, because the 52.16 series driver DOES NOT CHEAT (has no driver optimizations that break guidelines) in 3dmark03 patch 340.

Umm, that's possible I guess. I have a hard believing that considering the disparity between the results of 2 different builds. Without any comments from Futuremark I guess there is no way of knowing for sure.

No, he's saying that BECAUSE of the new patch, the new drivers aren't cheating specifically in 3DMark, because they can't. Without the patch to stop them, yes they're cheating - but that's irrelevant here.

exactly.
i fail to see how my statement could possibly have been taken to mean anything else.
Lets deconstruct it:
"the 52.16 series driver DOES NOT CHEAT in 3dmark03 patch 340".
seems pretty unequivocable. Cheating woudl be performance boosting, and patch 340 breaks the cheats for 330, ergo, the driver CANNOT be cheating in 340. If i sound pissed, i kinda am, because i stated things pretty clearly - if you can come up with another RATIONAL (and based on english usage) meaning, please, show me.

Man, settle down. I didn't mean to sound like I was flaming you, etc. If your post meant that, then fine, but your reply didn't make much sense because I already said that it was possible they started a clean slate and the guidelines were irrelevant to the current drivers. I've already pretty agreed to that theory since your post. See my replies to Joe.

Next time remember to take your meds on time. LOL j/k

Tommy McClain

 

[Xmas]

I'm interested in what exactly FM changed in the new build. If they changed some of the rendering code, it's not certain that the performance drop can be wholly attributed to circumventing application-specific optimizations.

 

[Bouncing Zabaglione Bros.]

i fail to see how my statement could possibly have been taken to mean anything else.
Lets deconstruct it:
"the 52.16 series driver DOES NOT CHEAT in 3dmark03 patch 340".
seems pretty unequivocable. Cheating woudl be performance boosting, and patch 340 breaks the cheats for 330, ergo, the driver CANNOT be cheating in 340. If i sound pissed, i kinda am, because i stated things pretty clearly - if you can come up with another RATIONAL (and based on english usage) meaning, please, show me.

I'm not arguing with what you intended to say, but I think it's worth pointing out that Nvidia may not successfully cheat in 340, but they do *attempt* to cheat. In 330, Nvidia attempts to cheat and succeeds. After seeing these cheats, Futuremark blocked them in 340, so that the same cheats now fail. In both cases the *intention* and code from Nvidia is to artificially inflate the benchmark scores.

You may say the Nvidia do not cheat on 340 because they are not able to, but this is not due to the honesty or intention of Nvidia, but the (currently) successful attempts at policing by Futuremark.

An analogy would be that Nvidia walk into an exam carrying a cheat sheet. In the 330 exam, Nvidia get away with the cheating and get high scores on the exam that they do not deserve. In the 340 exam, the ajudicator (Futuremark) sees the cheat sheet, and takes it away from Nvidia. Nvidia are then allowed to continue the exam without the cheat sheet, and come out with a lower, honest score.

Several people suggest (myself included) that for Nvidia to go into that exam intending and equipped to cheat means they should be disqualified from taking the exam when they are discovered cheating. Futuremark's approach is to simply confiscate the cheat sheet and allow Nvidia to continue.

Of course all of this assumes that Futuremark has indeed caught every Nvidia cheat, and that there are not more of them undiscovered and hidden in Nvidia's encrypted drivers artificially inflating their 3DMark score.

 

[CorwinB]

Or, you are missing the boat, because the 52.16 series driver DOES NOT CHEAT (has no driver optimizations that break guidelines) in 3dmark03 patch 340.

Seems to me that someone is probably missing the shore here...

From the results, it looks like the 52.16 drivers cheatin 3DMark, and do not comply with FM's rules. The cheats are then made irrelevant by patch 340. Or do you think there is a specific code path in the driver that says "If 3DMark version > 330, disable cheat" ?

 

[AzBat]

F*ck you Tommy! How dare you disagree with me!

ROFL Good thing we're friends! Just needed to remind you that I won't always be there to agree! So be careful the next time you get in yelling match. I may not be there to back you up! :lol!

The time that passes between the release of a NVIDIA driver that cheats and when they can release a driver to defeat it could get really long. So there is already going to be improper use of their benchmark during that time.

Good point.

See? I can think for myself most of the time. Seriously... Wait a minute, why aren't you laughing?

But according to their guidelines NVIDIA will be forbidden to use the cheating drivers with 3DMark03 in any official capacity. Plus, Futuremark themselves will not use the drivers in the Hall of Fame, etc. If web sites start using them in reviews(against the license agreement) I would hope people would notify Futuremark about it and let them handle them.

Another good point...although this raises the "who will use 3DMark then?" quesiton. In other words, if FM starts threatening web sites with legal action, are web sites going to bother giving FM support?

I didn't say they needed to threaten with legal action. Just a friendly reminder that they are doing a disservice to Futuremark and their readers by not using the approved drivers for 3DMark results. That should be sufficient I would think.

If we start seeing Futuremark releasing a patch every time a cheating driver is released then nobody will use the benchmark because Futuremark is basically saying all versions are invalidated.

No, that's not true. All results from patch 340 and the "currently approved" drivers will be valid, now and forever. Once (if) a new patch is released, those older results will not go away, and FM can also still accept results from patch 340 with the "patch 340 approved hardware/drivers".

Ok, you got a point there. But I still think that if they release newer patches it gives a bad appearance to end-users such that they believe the results can't be trusted.

It's a good thing we're not keeping score.

Tommy McClain

 

[AzBat]

I'm interested in what exactly FM changed in the new build. If they changed some of the rendering code, it's not certain that the performance drop can be wholly attributed to circumventing application-specific optimizations.

I'm interested in it as well. Their press release isn't very forthcoming. If we knew what they changed, then maybe that would tell us why NVIDIA's score is lower. Unfortunately I don't think Futuremark is going to tell us either of those issues.

Tommy McClain

 

[OpenGL guy]

I'm interested in what exactly FM changed in the new build. If they changed some of the rendering code, it's not certain that the performance drop can be wholly attributed to circumventing application-specific optimizations.

If the changes were significant, then the Radeons would show differences as well.

 

[Som]

If Nvidia cares enough to cheat, I care to know about it. This has been gone over a million times but OEMS use scores like 3dmark to choose cards.

Well, firstly, if someone buys an OEM PC, they deserve what they get. 'Nuff said.

An individual game test will tell you about 1 game, 3dmark is attempting to show probable results of the hardware in all games. Also game benchmarks aren't 'policed' by the game developers. Futuremark has a vested interest in maintaining the integrity of the results for 3dmark.

So far 3dmark has been a fairly accurate predictor once you weed out the cheats. I see no reason why it can't continue to do so.

All good and well, but you're all still missing the point:

If the performance is there, who cares why?

Don't mistake me for some Nvidia fanboy - I wouldn't be buying an NV card right now if I were upgrading, but I don't see the problem in different manufacturers using different driver code to achieve similar or identical results.

As I said, we had numerous thorough reviews from sites like Anandtech and Hard OCP analyzing performance if Nvidia and ATi architecture in many many games, sometimes 20+, across DX8 and DX9.

In short, we know what these cards are capable of, no 3dMark needed. If Nvidia is using "cheat code" to render the same things ATi renders, and performance and IQ are the same, then what does it matter?

Of course, the most obvious response would be that if they're rendering the same thing it's not a cheat - connotating that Nvidia isn't rendering the same stuff as ATi when 3dMark is run, under which circumstances I request evidence. Show me this alternate stuff being rendered that is resulting in a higher score.

IIRC, as of the Forceware release, aside from some fishy Brilinear typ filtering on the FX line, Nvidia and ATi cars had identical IQ when you leave AA and AF out of the picture - in other words, the same frames were being rendered by both company's cards.

Feel free to correct me if I'm wrong somewhere in here. I'd certainly like to know - it is not by any means out of the realm of possibility that I'm ignorant of one analysis or another. I don't have all day to spend snooping the internet.