Futuremark Announces Patch for 3DMark03

The way I see it is that FM had to stop cheats from IHV drivers by changinf their code and performing counter actions. nVidia or ATI sure would not start this if they were just asked to. This way FM can show that there is funny things going on and they don't humiliate IHVs but just show the result.

This leads to exactly what Joe wrote: nVidia has time till next WHQL drivers to get rid of the cheats. And it's also FM's responsibility to not approve any of the newer drivers if they require patching of 3DMark03 to get rid of cheating.

Of course this could lead to very good cheats that are not detected easily. We can hope and vote with our wallet that it won't go there.

Som: I used to think 3DMark03 is waste of time when they changed their statement from cheating to optimization. But now FM is pulling their act together and it seems that results from 3DMark03 can be compared again. It has shown already results of nVidia's and ATI's graphics cards that are in line with those few DX9 games.

 

Actually there are some Complants even in "real World Tests".

First post... Interesting timing on your Arrival and the specific opinion path and view of the information

 

If Nvidia cares enough to cheat, I care to know about it. This has been gone over a million times but OEMS use scores like 3dmark to choose cards.

An individual game test will tell you about 1 game, 3dmark is attempting to show probable results of the hardware in all games. Also game benchmarks aren't 'policed' by the game developers. Futuremark has a vested interest in maintaining the integrity of the results for 3dmark.

So far 3dmark has been a fairly accurate predictor once you weed out the cheats. I see no reason why it can't continue to do so.

 

I agree, because it didn't look like that to me.

Tommy McClain

 

"So, if Nvidia is running "non-standard" code (like that exists or something) in the background to achieve their results, WHO CARES? We know what we're getting when we buy, regardless of whether or not we truly know how we get it."

This is the second time in 10 minutes I've seen this same assignine argument used by a one-post-wonder. I was gonna call you 'paranoid', but...

(The Dig pulls out his trusty tinfoil hat and carefully unfolds it and places it firmly on his head making sure it completely encases his brain.)

 

There will be another Futuremark patch following the next driver release from Nvidia, so once we figure out how many Nvidia drivers releases are left this year, we can then extrapolate that into how many patches will be released from Futuremark

 

I care, and no, we don't know what we're getting if nVidia is using app specific code to prop up benchmarks.

Why?

Because you cannot tell me that nVidia will put app specific code in their driver for every single application out there. At best, nVidia could put app specific code for popular games and for benchmarks.

So what happens when I have a game that has no "special app detection" in the drivers?

1) You end up having to wait for a new nVidia driver release to play the game with expected performance

or

2) You never get drivers updated for that game.

This is one of the most important threads wrt drivers and cheating in quite long time.

Unfortunately, that doesn't help someone who's buying a new card NOW, does it? I can be buying a card to play my EXISITNG games better...this doesn't mean I shouldn't consider the possibilities with future games.

In other words, if Card A and Card B both play "current games" very well....but "future benchmarks" (like 3DMark, aquamark, Shader Mark, etc.) show that Card As performance and quality is solid and predictable, and Card B is "flaky...depends wildly on drivers and acceptance of certain quality degredations"....

What card would you buy?

 

Hi Tommy!

I understand what you're saying, but I still think 3DMark will need to release patches....because even though scores won't be officially submitted to the DataBase for non approved drivers, that won't stop reviewers from using "the latest nVidia drivers" and testing them with 3DMark in their reviews. (Against the license agreement.)

It's just a safeguard against improper use of their benchmark.

My only disagreement here, is that they should issue a patch, but the newer drivers should not be approved.

 

The time that passes between the release of a NVIDIA driver that cheats and when they can release a driver to defeat it could get really long. So there is already going to be improper use of their benchmark during that time. But according to their guidelines NVIDIA will be forbidden to use the cheating drivers with 3DMark03 in any official capacity. Plus, Futuremark themselves will not use the drivers in the Hall of Fame, etc. If web sites start using them in reviews(against the license agreement) I would hope people would notify Futuremark about it and let them handle them.

If we start seeing Futuremark releasing a patch every time a cheating driver is released then nobody will use the benchmark because Futuremark is basically saying all versions are invalidated. Who's going to want to buy and use 3DMark then? Nobody.

Tommy McClain

 

exactly.
i fail to see how my statement could possibly have been taken to mean anything else.
Lets deconstruct it:
"the 52.16 series driver DOES NOT CHEAT in 3dmark03 patch 340".
seems pretty unequivocable. Cheating woudl be performance boosting, and patch 340 breaks the cheats for 330, ergo, the driver CANNOT be cheating in 340. If i sound pissed, i kinda am, because i stated things pretty clearly - if you can come up with another RATIONAL (and based on english usage) meaning, please, show me.

 

F*ck you Tommy! How dare you disagree with me!

Good point.

Another good point...although this raises the "who will use 3DMark then?" quesiton. In other words, if FM starts threatening web sites with legal action, are web sites going to bother giving FM support?

No, that's not true. All results from patch 340 and the "currently approved" drivers will be valid, now and forever. Once (if) a new patch is released, those older results will not go away, and FM can also still accept results from patch 340 with the "patch 340 approved hardware/drivers".

 

I think as long as the OEMs see a benefit to using 3dmark and see that the field is as level as possible it will be ok for FM.

 

Man, settle down. I didn't mean to sound like I was flaming you, etc. If your post meant that, then fine, but your reply didn't make much sense because I already said that it was possible they started a clean slate and the guidelines were irrelevant to the current drivers. I've already pretty agreed to that theory since your post. See my replies to Joe.

Next time remember to take your meds on time. LOL j/k

Tommy McClain

 

I'm interested in what exactly FM changed in the new build. If they changed some of the rendering code, it's not certain that the performance drop can be wholly attributed to circumventing application-specific optimizations.

 

I'm not arguing with what you intended to say, but I think it's worth pointing out that Nvidia may not successfully cheat in 340, but they do *attempt* to cheat. In 330, Nvidia attempts to cheat and succeeds. After seeing these cheats, Futuremark blocked them in 340, so that the same cheats now fail. In both cases the *intention* and code from Nvidia is to artificially inflate the benchmark scores.

You may say the Nvidia do not cheat on 340 because they are not able to, but this is not due to the honesty or intention of Nvidia, but the (currently) successful attempts at policing by Futuremark.

An analogy would be that Nvidia walk into an exam carrying a cheat sheet. In the 330 exam, Nvidia get away with the cheating and get high scores on the exam that they do not deserve. In the 340 exam, the ajudicator (Futuremark) sees the cheat sheet, and takes it away from Nvidia. Nvidia are then allowed to continue the exam without the cheat sheet, and come out with a lower, honest score.

Several people suggest (myself included) that for Nvidia to go into that exam intending and equipped to cheat means they should be disqualified from taking the exam when they are discovered cheating. Futuremark's approach is to simply confiscate the cheat sheet and allow Nvidia to continue.

Of course all of this assumes that Futuremark has indeed caught every Nvidia cheat, and that there are not more of them undiscovered and hidden in Nvidia's encrypted drivers artificially inflating their 3DMark score.

 

Seems to me that someone is probably missing the shore here...

From the results, it looks like the 52.16 drivers cheatin 3DMark, and do not comply with FM's rules. The cheats are then made irrelevant by patch 340. Or do you think there is a specific code path in the driver that says "If 3DMark version > 330, disable cheat" ?

 

ROFL Good thing we're friends! Just needed to remind you that I won't always be there to agree! So be careful the next time you get in yelling match. I may not be there to back you up! :lol!

See? I can think for myself most of the time. Seriously... Wait a minute, why aren't you laughing?

I didn't say they needed to threaten with legal action. Just a friendly reminder that they are doing a disservice to Futuremark and their readers by not using the approved drivers for 3DMark results. That should be sufficient I would think.

Ok, you got a point there. But I still think that if they release newer patches it gives a bad appearance to end-users such that they believe the results can't be trusted.

It's a good thing we're not keeping score.

Tommy McClain

 

I'm interested in it as well. Their press release isn't very forthcoming. If we knew what they changed, then maybe that would tell us why NVIDIA's score is lower. Unfortunately I don't think Futuremark is going to tell us either of those issues.

Tommy McClain

 

If the changes were significant, then the Radeons would show differences as well.

 

Well, firstly, if someone buys an OEM PC, they deserve what they get. 'Nuff said.

All good and well, but you're all still missing the point:

If the performance is there, who cares why?

Don't mistake me for some Nvidia fanboy - I wouldn't be buying an NV card right now if I were upgrading, but I don't see the problem in different manufacturers using different driver code to achieve similar or identical results.

As I said, we had numerous thorough reviews from sites like Anandtech and Hard OCP analyzing performance if Nvidia and ATi architecture in many many games, sometimes 20+, across DX8 and DX9.

In short, we know what these cards are capable of, no 3dMark needed. If Nvidia is using "cheat code" to render the same things ATi renders, and performance and IQ are the same, then what does it matter?

Of course, the most obvious response would be that if they're rendering the same thing it's not a cheat - connotating that Nvidia isn't rendering the same stuff as ATi when 3dMark is run, under which circumstances I request evidence. Show me this alternate stuff being rendered that is resulting in a higher score.

IIRC, as of the Forceware release, aside from some fishy Brilinear typ filtering on the FX line, Nvidia and ATi cars had identical IQ when you leave AA and AF out of the picture - in other words, the same frames were being rendered by both company's cards.

Feel free to correct me if I'm wrong somewhere in here. I'd certainly like to know - it is not by any means out of the realm of possibility that I'm ignorant of one analysis or another. I don't have all day to spend snooping the internet.