ExtremeTech Article Up

[TempestX]

Driver Irregularities May Inflate nVidia Benchmarks

http://www.extremetech.com/article2/0,3973,1086857,00.asp

If this needs to be moved to the 3D Graphics Boards and Drivers please do so.

 

[MuFu]

http://www.beyond3d.com/forum/viewtopic.php?t=5833&start=40
http://www.beyond3d.com/forum/viewtopic.php?t=5840

I guess this warrants its own thread though.

MuFu.

 

[TempestX]

Thx MuFu,

Edit: Dave or Reverend never mind. LOL

 

[John Reynolds]

Nah, this one deserves its own thread, though I fear it'll turn into a flame fest. Let's try to keep it clean.

 

[WaltC]

Well, I've already said my piece as related in one of Mufu's links above. Not much else to say, really, except that I'm not in the least surprised.

 

[Archaeolept]

so, how plausible is it that this might actually be some sort of unintentional bug, as Nvidia claims?

it sure doesn't look like it, but I'm one of the ignorant multitude, so have no real idea.

 

[John Reynolds]

so, how plausible is it that this might actually be some sort of unintentional bug, as Nvidia claims?

it sure doesn't look like it, but I'm one of the ignorant multitude, so have no real idea.

What the article doesn't point out is that the anomalies have been getting worse with each new driver update. Anomalies increase = increased score. What's scary to me is that the same thing could be done for a game's timedemo.

 

[Humus]

One thing is for sure. NVidiots will defend nVidia to death claiming it is a just driver bug (yeah, right!), just like fanATIcs did for ATI. We have now seen cheats from ATI, nVidia and Trident. Excuse me while I go crying in a corner

 

[WaltC]

so, how plausible is it that this might actually be some sort of unintentional bug, as Nvidia claims?

it sure doesn't look like it, but I'm one of the ignorant multitude, so have no real idea.

Running the numbers on my pocket calc I come up with odds of 1,467,786,877,956 to 1 against this kind of thing resulting from an unintentional driver bug.

 

[Archaeolept]

^^lol

kinda what I thought. are there ways to check out other timedemos than 3dmark2k3 for similar "bugs"?

and, of course, the big question is what could have been done to jink Doom3.

right when I thought Nvidia had pulled a real rabbit out of it's hat.

 

[MuFu]

What the article doesn't point out is that the anomalies have been getting worse with each new driver update. Anomalies increase = increased score. What's scary to me is that the same thing could be done for a game's timedemo.

That's the key thing. I have gone off on one over at nV News regarding this so I'll try and keep my cool in here.

Two things bug me even more than the cheating - the fact that they have been lying to reviewers about this and the sheer hypocrisy of the whole thing, having given FutureMark so much stick about 3dm when it was first released.

MuFu.

 

[Nazgul]

One thing is for sure. NVidiots will defend nVidia to death claiming it is a just driver bug (yeah, right!), just like fanATIcs did for ATI. We have now seen cheats from ATI, nVidia and Trident. Excuse me while I go crying in a corner

And while I certainly found ATI's official response to the whole "Quack" thing somewhat fishy, one has to give them props for quickly coming out with a driver that not only fixed the problem, but did so with no loss of performance. It remains to be seen what the next set of DetFX drivers will do.

 

[OpenGL guy]

It remains to be seen what the next set of DetFX drivers will do.

My prediction: No loss in performance. Remember the "splash screen" debacle with 3D Mark 2001? It's the same thing all over again. Before the driver just changed how it detected the benchmarks, this time the driver will just disable the "optimizations" when needed.

nvidia has shown they are willing to invest a lot of time in these "optimizations", so they'll want to keep them. I guess people should have taken that white paper nvidia released about 3D Mark 2003 as a recipe for "optimizing" for a particular benchmark.

Trident getting caught by simply renaming the executable is just shameful I had a good laugh when I read about that.

 

[Sabastian]

That is interesting. I get the same sorts of tracers yet on my Geforce DDR in Unreal Tournament. It always seems to happen in a large open area with lots of sky on particular game maps. I don't know if it really means anything or not but I have always had that problem. No one seems to remmember how horrible the GeforceDDR drivers were and it seemed that they were bad for a long time. Dare I say it... I think nvidias drivers reputation is just a pile of BS. IMO of course.

http://common.ziffdavisinternet.com/util_get_image/2/0,3363,sz=1&i=24047,00.jpg

EDIT: Picture too large.

 

[Doomtrooper]

http://www.nvnews.net/articles/david_kirk_interview.shtml

We at NVIDIA don’t make it a practice to optimize our pipeline for specific benchmarks - we want to provide high quality and high performance on a wide variety of useful and entertaining applications, rather than just getting a good score. Ask yourself (or, better yet, ask ATI) why Radeon 8500 performs well on this one test, and poorly on many other 3DMark2001 tests.

David Kirk

 

[DadUM]

Well, nVidia has always been against 3DMark2k3. Seems like this might be a gigantic blow against it. nVidia's aim may either get the top score or invalidate the whole test. It now remains to be seen what the next moves will be. Look at the end of the Extreme Tech article:

Here at Ziff-Davis Media, we've dealt with this problem for over 10 years, on our popular 3D, system, and networking benchmarks. Cheating is one reason why we've always been in favor of benchmarks using real-world applications -- because optimizing those benchmarks also tends to optimize real-world performance. Synthetic benchmarks like 3DMark2003 have their place, but they are somewhat more susceptible to unfair optimizations.

nVidia may have just gotten what they wished for.

 

[Doomtrooper]

If your card is slow rendering Pixel Shader 2.0 in Nature, it will be slow rendering Pixel Shader 2.0 in S.T.A.L.K.E.R.

 

[micron]

Man....this discussion is the topic of forums across the web. This discussion here on B3D, I found linked on at least two or three different sites.

 

[WaltC]

...
nVidia may have just gotten what they wished for.

Actually, I think this sort of behavior will do more to invalidate nVidia than to invalidate the benchmark... All benchmarks are "susceptible" to manipulation--all of them. IMO, companies will be tagged as those who manipulate them and those who do not...

nVidia wasn't "fired" from the 3DMark program, nVidia voluntarily resigned from it. To blatantly fix your drivers to cheat it, and then to decry the fact that the cheat is a bug that resulted from the fact that you aren't in the program any longer, is both dishonest and hypocritical.

 

[Mark]

Inserting clip-planes into a popular benchmark, but only behind the viewport where it can't be seen, and where it coincidentally has huge benefits to performance ... that's a nice driver bug!