HardOCP's position on the 3DMark2003/Nvidia issue

[Slides]

http://www.theforumisdown.com/uploadfiles/0103/hardocpFX.jpg

 

[duncan36]

The random benchmark improving bugs strike again for Nvidia. WILL THE HORROR NEVER CEASE?

 

[demalion]

My perspective on Open Source benchmarks again, from earlier in the thread, since some of what I discussed seems relevant again.

 

[Pete]

Getting more on topic - If nVidia can detect that a predictable benchmark is being run and turn on hacks, they can just as easily detect when the camera is going in areas the driver didn't expect, and turn off the "optimizations"; making it nearly impossible to detect the hacks (sure you might see a slight framerate drop when it happens, but that could just as easily be written off as a change in scene complexity or a slight framerate dip due to a background process.. it probably wouldn't be any more than 2-5 fps). Maybe they had that functionality in the drivers and it didn't work, hence nVidia calling it a 'bug'?

I think the framerate drop would be more noticable. If this hack only gained nVidia a few fps, I'm not sure they would've bothered with it.

 

[g__day]

Hey dude - where's my dragon????

http://www.nvchips-fr.com/images/news/madonion/draghotic_s.jpg

Its not just 3d Mark 2003 - 2001 dragothic is affected too, under certain conditions the dragon dissappears!!!

http://translate.google.com/translate?u=http://www.nvchips-fr.com/News/news.php?i

/couldn't_resist

 

[Kristof]

The dragon issue might be caused by using DirectX in debug mode rather than retail... not sure though but worth a check if you have the SDK and 3DM2001 installed. Debug does more checking and can fail draw primitive calls while retail version does not do so many checks and would not fail the draw primitive call. Just a guess though...

K-

 

[blake_mooney]

I'll just repost what I wrote in another thread where this was brought up..I do think this deserves a thread of it's own:

I just read the blurb on [H] and I am pretty sickened by it. Right from the first senetence:

"Two days after Extremetech was not given the opportunity to benchmark DOOM3, they come out swinging heavy charges of NVIDIA intentinoally inflating benchmark scores in 3DMark03...."

So, right off the bat they make the implication that ExtremeTech has some sour grapes becuase they didn't get Doom3? So while H gets pats itself on the back and says ExtremeTech is unfounded in its opinion, it has no problem laying the foundation for their own perceived motives of ExtremeTech?

HARDOCP HAS SUNK TO A NEW LOW. I should say, that specifically KYLE has sunk to a new low, as he made the update.

I can play that game too, Kyle

"Two days after HardOCP was given favorable treatment by nVidia with a Doom3 benchmark, they refuse to acknlowedge or investigate legitimate data that shows improper behavior in an industry standard benchmark."

Rather than debunk ExtremeTech's research (which would be a proper retort), they resort to accusations of sour grapes motives.
Which, [even if that were a 100% true motivation, doesn't make the data any less legitimate.

And who didn't see this coming, as predicted by many already:

"The first thing that came to mind when I heard about this, was to wonder if NVIDIA was not doing it on purpose to invalidate the 3DMark03 scores by showing how the it could be easily manipulated."

Hey Kyle, apparently, it's NOT so easy to manipulate it, and not get caught.

thats screwed up [H] is so full of shit

 

[K.I.L.E.R]

"But if you can't SEE the cheating, isn't that an optimization? - Kyle Bennett ([H]ard|OCP)

cheat ( P ) Pronunciation Key (cht)
v. cheat·ed, cheat·ing, cheats
v. tr.
To deceive by trickery; swindle: cheated customers by overcharging them for purchases.
To deprive by trickery; defraud: cheated them of their land.
To mislead; fool: illusions that cheat the eye.
To elude; escape: cheat death.

Why does he bring them together?

I'm a coder and I can tell that a cheat is a cruddy shortcut used to achieve something. In my case the desired result temporarily so I won't have to do much work but I eventually found it to be a fruitless attempt as the teacher looks at my code and then I do more than twice the amount of work.

Optimisations are a balance between storage and processing.

It's obvious Kyle knows very little about software development.

 

[digitalwanderer]

Did anyone ask Kyle yet?

Just in the interests of fairness and all, I was just curious. I sent him an e-mail a little while ago, and posted this in his forum...but has anyone actually heard a response from Kyle yet? (You're excluded from this one Brent, as I can dig you not being able to talk about it. )

 

[T2k]

Ehehehemmm...

[H]ardNews 5th Edition Friday May 23, 2003
Posted by Steve 10:50 AM (CDT)

3DMark Patch:
Futuremark has released a patch for 3DMark 2003 that eliminates “artificially high scoresâ€￾ for people using NVIDIA Detonator FX drivers. This is in response to the news item we posted last week ( as did several sites ). According to the PDF on Futuremark's site, the patch causes a 24.1% drop in score for NVIDIA and a 1.9% drop in score for ATi meaning NVIDIA isn't alone in this whole ordeal, other companies have "irregularities" as well. It is good to see Futuremark fix the current situation to remove any unfair advantages, but this goes to show you what we meant in our "Benchmarking Right" article. Thanks to everyone who sent this one in.

Really preposterous.

These guys... ahh, forget them.

 

[saf1]

Really preposterous.

These guys... ahh, forget them.

I'll finish it - SUCK!

 

[Ostsol]

[H] is missing a very important point that FutureMark made in the audit document: that it is almost impossible to detect driver cheats in games. If there is a visible quality difference, then it's easy, but if the difference is in time demos and like what NVidia did in 3dMark03, it's impossible to tell. Also, game developers have neither the time nor the inclination to go hunting for driver cheats. All they're concerned with is the compatibility of their game with the hardware it may potentially be played on. A company like FutureMark, however, must be dedicated to the accuracy of their product in analyzing the rendering performance of a video card. They certainly do have the time and inclination to ensure that all video cards are rendering the benchmarks correctly.

 

[digitalwanderer]

Well, I tried.

I give up!

Dang it, I tried talking sensibly to those folks...but I always end up losing it when they start using such great logic as, "But it's a benchmark, so what?"

I give up, [H] is hopeless again.