HardOCP's position on the 3DMark2003/Nvidia issue

[MuFu]

I can't help but think that if they had been privy to this info last week they'd have published a similar article to ET's. Their inconsistency has been pretty shocking in the past.

The idea of a "fair benchmarking" initiative is good one but surely 3DMark is just an evolution of the very same principles?! I can't believe this issue has been misconstrued by so many to be that of whether 3DM is a valid benchmarking tool or not, instead of one of IHV morality when it comes to bench-specific optimisations.

There will ALWAYS be ways of optimising for a particular benchmark so a certain amount of trust has to exist between IHVs, reviewers and consumers. When this trust is betrayed the main focus should be on resolving the cheats, not using them to invalidate the benchmark! Tools such as 3DM are invaluble for OEMs, AIBMs etc... getting rid of them won't solve anything because similar solutions will rapidly pop up to take their place.

Kyle may think he is on some kind of moral crusade, but it is terribly misguided. He is barking up the wrong tree completely.

MuFu.

 

[Joe DeFuria]

So ET didn't assume nvidia's motivation was higher 3dm2k3 scores (cheating)? Like I said, it doesn't matter... a conclusion is still a conclusion - not a fact.

Um, the motivation for a card company to get a higher score is obvious...higher scores make your card look better = more sales. "Cheating" is not a motivation. It's a means to the end.

ET concluded that, based on evidence that they presented and published, they believe nVidia is cheating. They DID also contact nVidia to get their initial opinion on the matter, and relayed nVidia's message to the readership.

KYLE presented some "rumors" that ET was not happy about being left off the Doom list, as some motiviation for publicizing the article.

No facts to back that up (like an e-mail from someone at ET?). Nor did Kyle contact ET about what their motivation actually, and relay that message to his readership.

The two situations are not remotely similar.

EDIT: It doesn't matter if one is 99% likely to be true and the other 1%... both are unverifiable and thus not representative of the truth.

Wrong.

ET provided us their own evidence, PLUS a verification from other sources that validate the evidence is ligit. This doesn't mean nVidia is "purposely cheating", but the evidence is in full support of that conclusion pending further results.

Kyle has presented NOTHING to either refute that conclusion, or assert his own theory for the motivation behind the article, other than "his word."

EDIT2: I like how everyone accusses Kyle of assuming motivations but then pretend to know Kyle's motivation... that's pretty funny heh.

We here are commentators on a web forum. We are not running a web site that is slamming another web site that is in competition with us.

 

[Evildeus]

If you're referring to the developer build of 3DMark2K3 that allows free camera movement, It's stated quite clearly that Futuremark authorized the use of the screenshots on the front page to illustrate the problem, which would indicate that they've been made aware of the situation. If they had any objections to the material being made public, I imagine they'd have said something to Extremetech/B3D by now.

Sure, but:
- You are a beta member, if it was Ati, how would this issue be taken?
- Doesn't that suit them? They are not taking the risk, you are having a scoop.
- "they've been made aware of the situation" I hope so. Then how did this issue araise?
- If it was a particuliar beta member who had the scoop, do you think it would be taken as it is now?
- Even if they have some objections now, it's too late, because the consequences are here, and the objectives have been reached.

Looking at this affair, it seems that ET/B3D benefits from the scoop, Futuremark (and the other beta members) from not being involved, [H] from being the Nv Angel, almost everybody seems happy.

 

[ninelven]

A conclusion is a conclusion, but let's use a little logic and sound reasoning, shall we?

By all means....

I'm perfectly aware of all the data one might draw upon to reach said conclusions; I did not say which one I believed.

Either way, it doesn't matter what either I or anyone else believes because its not going to change the fact that either A) nvidia cheated or B) they did not cheat. A conclusion/reality I am sure we will all be better able to discern as time progresses.

Edit:

We here are commentators on a web forum. We are not running a web site that is slamming another web site that is in competition with us.

Please don't tell me that moral reponsibility has something to do with one's position in society.... one can make anything seem insignificant if he just applies his reason for long enough.

 

[Deflection]

Kyle makes a conclusion based upon analysis of data as he sees it and then gets slammed for doing so? One certainly has a right to disagree with his ideas, but to question his right to provide this analysis is largely hypocritical.

Kyle made an analysis of assumed motivation.

So did ET. So what? It's perfectly within their rights.

I pretty much agree with ninelven. I don't see what there is to get all worked up about. I never once believed Nvidia didn't cheat/optimize on benchmarks when they could. I'm sure they'll do it again too Same goes for ATI. Just look at what OpenGL Guy said from this post.

My prediction: No loss in performance. Remember the "splash screen" debacle with 3D Mark 2001? It's the same thing all over again. Before the driver just changed how it detected the benchmarks, this time the driver will just disable the "optimizations" when needed.

nvidia has shown they are willing to invest a lot of time in these "optimizations", so they'll want to keep them. I guess people should have taken that white paper nvidia released about 3D Mark 2003 as a recipe for "optimizing" for a particular benchmark.

Trident getting caught by simply renaming the executable is just shameful I had a good laugh when I read about that.

 

[Jima13]

*sigh*

"Argument against the Person

Two arguers: One presents an argument and the other responds by redirecting attention away from the argument and towards the arguer. The question of whether the premises support the conclusion are ignored. But the merits of an argument are independent of the character of the arguer.

Abusive version: respondent verbally abuses the arguer and ignores the argument.

Circumstantial version: respondent calls attention to special circumstances of the arguer and ignores the argument.

Tu quoque: respondent attempts to make the arguer look hypocritical.

Sometimes the character of a witness or informant are relevant to the credibility of testimony.

Sometimes it is fair to consider the character of the arguer before we accept the premises of an argument.

The fallacy (against the person) occurs only when the person attacked is an arguer and when our attention is drawn from the character of the argument and to the character of the arguer."

Considering the argument 'is it a cheat or bug'.......

 

[Joe DeFuria]

So did ET. So what? It's perfectly within their rights.

How many times does this have to be stated?

Kyles "analysis" consisted of nothing but "rumor", with no attempt to ascertain the story from the "accused." ET's analyis consisted of a repeatable exercise verified by third parties, and he did contact the accused for "their story", and passed that on to the readers.

Just look at what OpenGL Guy said...

What did OpenGL guy say that made you believe ATI is in the same boat?

 

[Clashman]

There does seem to be a definate discrepency with how Kyle views the actions of one Hardware vendor versus another. Had they applied their own advice to the present situation, there is no reason to blow off what's happening at present like they did.

From QUACK

The Facts As We See Them:

It certainly seems to us here at [H]ardOCP that ATi has in fact included application specific instructions in their version 5.13.01.3276 Win2K drivers that make Quake 3 arena benchmarks faster by up to over 15%.



The Real Question:

Does this really make a difference?

There are those that will argue that ATi is in fact cheating at the benchmark game by producing and distributing drivers that are aimed at increasing scores in what is the most widely used 3D benchmark in the world. The other side of that argument is that ATi is simply building a driver set that is aimed at giving Quake 3 Arena players the smoothest gaming experience that ATi can deliver.

We are not going to interject a lot of opinion on this matter except to say this. We think that ATi should be producing Radeon drivers that are 3D engine specific and not game specific. Especially when the one targeted game is a widely used benchmark that people trust.

As we mentioned we are not out to start an anti-ATi crusade over this matter as the Radeon 8500 already has a great battle in front of it in the battle for market share. Still we think it is important in situations like this that you are armed with this type of information when evaluating your next VidCard purchase. ATi engineers were asked last week if ATi drivers used any game specific instructions and we were told "No."

In closing we would like to say that all the same testing was run on the latest set of NVIDIA DetonatorXP drivers without any of the same issues. Your thoughts and opinions are always welcome HERE with us as well.

 

[Evildeus]

Hellbinder,
I understand what you mean but i can read some of the guidelines of Futuremark:

Cooperation is always carried out with utmost confidentiality

http://www.futuremark.com/betaprogram/

It doesn't seems to be the case right now.

Nvidia is at fault, there's no doubt about it (i don't mind if it's a bug or cheat) but the way this information is exposed doesn't seem right also (from my point of view). Or perhaps we don't have the same notion of confidentiality.

Moreover i don't think ET could have done the same article using the same inside information on another beta member.

And yes, i don't like unethical behavior to expose unethical behavior. But that's just me

]Evil
The problem here is all this beta member talkof yours means absolutly nothing. I cant even fathom where you are comming from. WTH does being a beta member of Futuremark have to do at all with this? From an imperical evidence standpoint? There are no NDA issues here or anything else. These sites have ways to do detailed tests and others dont. its that simple. Just becuase *you* dont like the fact that this was exposed and made public does not suddenly make this some kind of ethics issue. Other than the cheater.

What.. No one is supposed to expose unethical behavior?? You are sounding like one of those lawyers who gets murderers off on some technicality. Or sues the family of the Victim becasue they supposedly were at fault somehow...

 

[Mariner]

As I understand things, HardOCP along with B3D, ExtremeTech, Anandtech, FiringSquad, Toms etc etc, are all thought of as independent and therefore (in theory at least) impartial web sites who should report on any news fairly, giving both sides of the argument a fair crack of the whip. Sites such as NVnews and Rage3D, on the other hand are avowedly 'fan' sites and should therefore be free to shout from the rooftops support of their preferred vendor if they so please.

Beyond3D appear to have given a fair viewpoint stating the apparent anomolies that occur in 3DMark and the possible reasons for these problems. They have then said they await a response to these problems from NVidia. This is all fine - nowhere have any accusations of 'cheating' been made, although this is a possible inference.

ExtremeTech have gone one step further and actively accused NVidia of cheating. This is a step towards what a fan site might say and might therefore colour their impartiality, but they have asked NVidia to respond to their accusations.

HardOCP, on the other hand, have posted a news story which doesn't query the ExtremeTech article whatsoever. Nowhere in the story does HardOCP ask for NVidia's response to the anomolous results. Instead, we have a statement which rubbishes ExtremeTech, questioning their motives whilst not addressing the salient points of the article. I'm actually left wondering if NVidia PR would have changed a single word of Kyle's statement if they were defending themselves!

This, I believe, is why everyone is getting so riled about this particular comment from Kyle. Personally, I've thought that HardOCP has been pretty fair to both sides of the coin over the last year or so. This news 'story', on the other hand, leads me to question their (or at least Kyle's) motives and impartiality.

Disclaimer: I've never owned an ATI product. I am currently using an NVidia card (my 3rd) and I am due to upgrade my computer soon with an nForce motherboard. So I am fair and impartial. Honest!

 

[MuFu]

And yes, i don't like unethical behavior to expose unethical behavior. But that's just me

But the fact that FutureMark endorsed the exposé surely invalidates your argument. It HAS been kept confidential up till now and was only revealed with their approval. What do you think people have been talking about via PM all week?!

MuFu.

 

[Malfunction]

So did ET. So what? It's perfectly within their rights.

How many times does this have to be stated?

Kyles "analysis" consisted of nothing but "rumor", with no attempt to ascertain the story from the "accused." ET's analyis consisted of a repeatable exercise verified by third parties, and he did contact the accused for "their story", and passed that on to the readers.

Just look at what OpenGL Guy said...

What did OpenGL guy say that made you believe ATI is in the same boat?

He is simply implying that ATi is capable of doing the same things.

It would require too much time/manpower to go through every driver and every benchmark used to figure out who was doing what when. Great it has been spotted, does it mean much to many... no.

Until Futuremark incorporates better versions of the tools or similar (FSAA viewer, FSAA tester, Texture Filter TestApp and Shadermark) into a version of 3DMark will a timed demo mean anything because it can always be optimized.

It's like watching a movie 300 times and being able to recite the lines. Countinuesly optimizing what, a demo for what purpose.. for a better score that means nothing?

More work on programs like FSAA viewer, FSAA tester, Texture Filter TestApp and Shadermark should be the focus, not a timed demo.

imho...

 

[Joe DeFuria]

HardOCP, on the other hand, have posted a news story which doesn't query the ExtremeTech article whatsoever. Nowhere in the story does HardOCP ask for NVidia's response to the anomolous results. Instead, we have a statement which rubbishes ExtremeTech, questioning their motives whilst not addressing the salient points of the article. I'm actually left wondering if NVidia PR would have changed a single word of Kyle's statement if they were defending themselves!

Bingo.

 

[jjayb]

I'm looking around at everyone rushing to defend Nvidia. Where were you when trident got busted last week? Where was Kyle's full page rant in defense of Trident? How many of you defending Nvidia defended Trident? How many of you dismissed the Trident cheating as "it doesn't matter it's only a benchmark"?

 

[Tahir2]

Mariner I think that was a very astute post you made there.

Basically don't shout about ethics and rubbish a site when in the recent past (twice) you have been caught out yourself. Your own credibility goes down to zero.

 

[Dave Baumann]

Looking at this affair, it seems that ET/B3D benefits from the scoop, Futuremark (and the other beta members) from not being involved, [H] from being the Nv Angel, almost everybody seems happy.

Believe me nobody benefits from this situation. Quite frankly I hate that we have been put into this sutuation.

From our part we have played it completely straight - I saw this a week ago and I sent both screenshots to NVIDIA (something which did actually get me into a little hot water with FM seeing as they are supposed to NDA'ed) and I've asked them to provide an explaination, to which they haven't as yet. What else are we supposed to do?

 

[Dave Baumann]

Beyond3D appear to have given a fair viewpoint stating the apparent anomolies that occur in 3DMark and the possible reasons for these problems.

Have we actually proffered any explainations yet? I've been steering clear of that personally, other than offering the facts of what we've seen.

 

[Tahir2]

I suppose this could blow your chances of getting an NV35 direct from NVIDIA. I am surprised you made that decision DaveB, as you could have pissed two parties off with one shot

I hope, at least, FutureMark and yourselves still have a good relationship. As for NVIDIA, well if they won't give you any products you do have other sources open for you

 

[Evildeus]

And yes, i don't like unethical behavior to expose unethical behavior. But that's just me

But the fact that FutureMark endorsed the exposé surely invalidates your argument. It HAS been kept confidential up till now and was only revealed with their approval. What do you think people have been talking about via PM all week?!

MuFu.

I'm sorry, but futuremark where did futuremark said they endorsed it? They didn't say anything just they are aware and investigating.

Futhermore, i hope you are wrong, because then ET/BED are becoming the spokesmen of Futuremark.

FutureMark, the developer of 3DMark2003, has told us that it is also aware of the issue with nVidia's 44.03 driver, and is currently investigating it as well.

http://www.extremetech.com/article2/0,3973,1087269,00.asp

 

[Malfunction]

I'm looking around at everyone rushing to defend Nvidia. Where were you when trident got busted last week? Where was Kyle's full page rant in defense of Trident? How many of you defending Nvidia defended Trident? How many of you dismissed the Trident cheating as "it doesn't matter it's only a benchmark"?

Their are not a whole lot of owners of Trident videocards that find it worth defending?

I will defend the validity of a benchmark when it actually means something while playing a game, DVD or working on 3D. If it is faster without sacrificing Image Quality for performance... terrific. If it is sacrificing Image Quality for performance in a game, DVD or 3D Program will I personally raise hell and be annoyed.

Till then...