Will Tom's, Anand, and other sites carry the 3DMark story?

[MrGaribaldi]

A Norwegian hardware site (Hardware.no) has posted this...

Here's a translation for those of you who can't read Norwegian (the majority I'd guess):

Futuremark, who are behind the popular benchmarks in the 3dmark series, has caught Nvidia modifying their drivers to achieve better performance in 3dmark03.

It all started when the popular site Extremetech published an article where they were suspicious of NVidia "cheating" in the their newest drivers. That because the performance was mcuh better in 3dmark03 than ATI, unlike what performance one saw in other tests.

Futuremark decided to take a closer look, and published friday the results in a document which proof of extensive optimizing in Nvidia's drivers, to increase the points achieved in 3dmark. By doing small changes in the pgroamcode, so the drivers didn't recognize 3dmark03, the performance dropped by with up to 25%.

Futuremark is now releasing an update for 3dmark03, which stops nvidia's drivers from recognizing the program, and thus avoiding "shortcuts" to increase the score.

This is not the first time a graphic chip producer is caught this way. ATI were caught earlier for optimizing their drivers with the intent to increase performance in a much used "timedemo"-test in Q3.

That graphic chip producers are optmizing their drivers for games comes as no suprise. It becomes a bit more serious when one takes shortcuts on purpose, and manipulate the drivers to increase the performance in benchmarks, which is supposed to be a neutral plattform to compare the performance of different cards. With regard for how widespread use there is of the 3dmark testprogram, it's not really suprising to see this kind of "cheating". Even though it in this case is only about a synthetic benchmark, a high score is undeniably important for marketing.

We recommend that one shouldn't base ones views solely on synthetic test when comparing graphic cards. Just as important is it to see the performance on a large specter of real applications. There isn't any reason for graphic cards producers not to optimize their drivers for these type of applications, but in the end it is performance and image quality in actual games and applications that matters, not some score produced in a synthtetic test.

There is no doubt that this is an unacceptable behaviour from Nvidia, but we can't say we're really suprised. We are waiting for the official statement from Nvidia - if they give one.

A quite balanced story imo, though still not taking in the possibility that Nvidia could be cheating in other programs/the importance placed upon 3dmark...

---
The forum is of mixed feelings, with some people "valiantly" defending Nvidia, some attacking 3dmark/synthetic benchmarks, and some (if not most) condemning Nvidia for cheating...
---

There was one thing pointed out in their forum, which strikes me as reasonable, though not likely to happen....

He said things was looking bleak for nvidia, but would wait with final judgement until other sources (not ET or B3D) could confirm FM's findings....

So if FM were to release a "special" version of 3dmark03 (with free flying camera) to more hardware sites, more people will take this seriously...

I'm not saying they should release this to everyone, but maybe give it to those sites who request it... Of course, they'd have to let the sites know that they could send such a request...

And to stop it from being released into the wild, they could maybe do something like this:
* Limited functions, (More or less the standard free version of 3dmark, but with the exception of not being able to publish results to the orb, and having the freeflying camera)
* The hardware sites would have to sign an agreement that they'd only use the special version for driver testing, and not release it into the wild.
* Hardcode the name of the hardware site into the program so any version suddenly showing up in the wild shouldn't be too hard to trace back to the original site..

Would something like this be feasible?

 

[Pete]

no_way, I think B3D have said (tip o' the hat to the Queen's English ) 43.51 shows (many of) the same opimizations. It should be obvious which drivers are optimized just by their scores compared to the original FX drivers, the 42.63. Note that scores jumped dramatically in the 42.68/9 drivers.

 

[Nagorak]

What I think will happen, sadly, is what happened at HotHardware. Despite the fact that FutureMark provided a mountain of evidence, and clearly outlined the case against nVidia, and despite the fact that nVidia has no response to the evidence, just unsupportable claims of purposeful wrongdoing by FutureMark, the sites will treat this as a "we can't really trust EITHER of them.

So they'll post benchmarks from both Pre and Post patch 330....not to show the difference or illustrate the effects of cheating, but to say "OK reader, you decide. We don't know who to believe."

Problem is, nVidia hasn't, at least yet, made ANY CASE for us to believe "their side" of the story. They haven't GIVEN us their side of the story, despite having been informed over a week ago.

Sad...utterly sad.

I expect Anand, who doesn't use 3DMark at all, to just completely ignore it. Despite the fact that it's a hugely newsworthy story in and of itself.

Anand is supposed to have a "detonator comparison" or a "new detonator" review or something like that...and it's overdue. I have a hard time believing he just wouldn't say ANYTHING about it. Mu hunch is something like this: "We've heard something about several vendors cheating on 3DMark, which we don't use. That doesn't have any impact on us, so we'll just get on with it."

Sigh...

Uhh...this reminds me of some unsupported claims made by a certain world leader in real life... Based on your track record, I think maybe you should be defending Nvidia.

 

[Pete]

Can we keep politics out of this? This issue has enough mindless support as it is.

 

[Joe DeFuria]

Uhh...this reminds me of some unsupported claims made by a certain world leader in real life... Based on your track record, I think maybe you should be defending Nvidia.

Joe DeFuria wonders about which reality you're speaking of...

 

[Humus]

There is more than one reality?

 

[martrox]

There is more than one reality?

Remember, that the reality you see is colored by the tinted shades of your perception........