HardOCP's position on the 3DMark2003/Nvidia issue

[John Reynolds]

Russ,

Sure he was stating facts. But the manner in which he states them certainly implies an ulterior motive on Salvatore's part that itself implies a possibility of the entire story being falsified. Because if motive is otherwise not an issue here then what we're left with are the facts that Nvidia has indeed been caught red-handed pulling a few 'tricks' with a very popular, very widely downloaded synthetic benchmark. Introducing motive introduces the possibility of the details presented in the article as being falsified, fictionalized, or whatever you want to call it. A truly dishonest way of rebutting the article without actually rebutting its contents.

So if Kyle is interested in just the facts why the tone of his post? Indeed, ET's motives for publishing the story fail to be relevent at all for someone interested in just the facts. At least IMO.

Does this post mean I've become polarized to one camp?

 

[nelg]

Considering Ziff-Davis have more potenial revenue to lose by steping on Nvidia's toes than does [H] I have no doubts as to which site has more integerity.

 

[RussSchultz]

But the manner in which he states them certainly implies an ulterior motive on Salvatore's part that itself implies a possibility of the entire story being falsified

I don't think he suggested falsified at all. Ulteriour motive, yes. Falsified? no.

 

[Tim]

a) a bug in Nvidia's drivers

Highly unlikely, the probability that the driver renders everything on the predefined camera path, but fails as soon as leave it at is very unlikely. Also the “bugâ€￾ only turns up in 3dmark.

b) an intentional cheat in Nvidia's drivers

By process of ellimination, this seems to be only possibility

c) a bug in 3DM2K3

That only shows up using NVIDIA cards using specific drivers

d) an intentional hack in 3DM2K3

But why?

 

[Hellbinder]

Russ..

Though, quite honestly, everybody will already flock to their predisposed polarity, so it really doesn't matter what I think.

What really bothers me here is your aloof atitude when you yourself are clearly displaying the exact behavior you seem to be Commenting on. What really bothers me are when people try to *Claim* they have no predisposed polarity when its completely obvious they do.

Actually, it would seem to me he was stating facts.

2 days after the Doom3 preview, this article making accusations shows up--even though the information has been around for a week or so (apparantly). This is a verifiable to be true or false.
It is heavily rumored that ET was upset they weren't included in the list. This is verifiable to be true or false (though, I suppose what does 'heavily rumored' mean)

Case in point.. How can you even begin to say that what [H] is posting are facts??? I mean come on.. Did you ever consider that it just took them a little time to make sure before they went public? That it looks like they contacted Nvidia about it ahead of time?? Remember the Quak issue? [H] totally blindsided ATi at the order of Nvidia. What Extremetech has done is FAR more tame.

Now supposedly from your aloof *non biased* position you are going to insinuate that this was all driven because of revenge bit based on *Rumors*.. Thus you discount the evidence in favor of the *non biased* position of *rumorism* That makes Extremetech the evildoers..

 

[Nite_Hawk]

Russ:

I really don't have a problem with his first statement, that they've known about it for a week. That's fine, but I don't think it offers any validity to one viewpoint or another. I have a big problem with the second though. He pressents rumor as if it were fact with no sources. What does "heavily" mean? What are the rumors specifically? Who started them? Where is it taking place?

He could just as easily say "It is Heavily rumored that RussShultz and Nite_Hawk wen't on a shooting rampage thursday morning and that they may have been rather angry while shooting people, as I was told by Nite_Hawk that he gets angry sometimes."

It's completely baseless, but everything said could be true. It certainly makes us look like bad doesn't it?

Nite_Hawk

 

[antlers]

As I was saying in the other thread, there are a host of possible scenarios, and it's not obvious that the evidence supports one over the other.

1. High level management at Nvidia sanctioned cheating on 3DMark in the manner suspected.

2. Lower level employee at Nvidia cheated on the benchmark without management's knowledge.

3. Driver bug that was not detected by NVidia QC.

4. Driver bug that was detected but passed along in the hopes that people wouldn't notice.

5. High level management at 3DMark detected Nvidia cards and purposely introduced errors to descredit Nvidia.

6. Lower level employees of 3DMark did what is described in item 6.

7. Bug in 3DMark that only affects Nvidia hardware or software.

How would one ever know which is the case unless the source code to 3DMark was opened? A better approach would be the benchmarking suite being discussed:

http://www.beyond3d.com/forum/viewtopic.php?t=5853&start=0&postdays=0&postorder=asc&highlight=

The nature of the problem found pretty much precludes 3, 4 and 7 (could not happen by accident).

I don't see 3dmark doing 5 because it just makes them look bad, and they are trying to establish a benchmark. Purposely mucking up their benchmark undercuts their entire business model.

It would be pretty easy for 3dmark to rule out 6 now if that was any possibility. Although if 6 were the real explanation, 3dmark might be reluctant to admit it. In any case, NVidia has seemingly acknowledged their driver's responsibility for the problem.

That leaves 1 or 2. If it is 2, it shouldn't be very difficult for NVidia to figure out who is responsible. In that case they could come clean pretty quickly.

Given the amount of work involved to come up with such a cheat, however, I think 1 is the only reasonable possibility.


An open-source benchmark is stilll a good idea, though, probably based on one (or more) of the open-source game engines that are available. It's also clear that fixed-path, exactly repeatable benchmarks are useless in the face of unscrupulous IHV's, and Futuremark, if they want to continue to profit from 3d benchmarking, will have to address that.

 

[RussSchultz]

Hellbinder, before you go half cocked on me:

Is it true that 2 days after the Doom articles this one came out? Yes or no. Its a fact that can be verified to be correct or not.

Is it true that the cheating information has been around, before the Doom article went public? I don't know, but its either yes or no. In other words a fact.

Is it true that the article accused NVIDIA of cheating? Yes or no. Its a fact that can be verified to be correct or not.

It it true that it was rumored Extremetech was upset? I don't know, but it is either yes or no. In other words: a fact.

Christ almighty. Don't shoot me because you can't tell the difference between a fact and an opinion.

Some of you guys are talking like Kyle is just inventing shit and opinionating out his ass.


(Though, for the pedantics out there, I realize that facts are by definition true. However, nobody seems to be attacking the veracity of the statements, simply that the statements exist)

 

[Doomtrooper]

Kyle Bennett is not Qualified to even comment on the extremetech article, he doesn't even understand the article...he is not a programmer or a driver writer. He is a guy with a Webpage.

 

[demalion]

It seems that he just made a pretty strong implication that extremetech did this out of revenge.

Nite_Hawk

Actually, it would seem to me he was stating facts.

I understand your opinion, Russ, but you are contradicting yourself to propose it as invalidating Nite_Hawk's statement.

Here:

...
But, if those are facts, don't get yer panty's in a bunch that they're being reported. Admittedly editing by choosing the facts to present is a form of slanting the story.

Recognizing this is an effort for objectivity, but your initial proposition that things were other than Nite_Hawk's evaluation of a strong implication against ExtremeTech does not represent this same objectivity.

Though, quite honestly, everybody will already flock to their predisposed polarity, so it really doesn't matter what I think.

The result is you seem to display a predisposed polarity for yourself, even though stating some otherwise reasonable observations.

I make this observation because I view Kyle's commentary as a quite significantly larger scale version of the same occurrence, so it hopefully conveys my disagreement on all levels. Might be a bit recursive..so I'll illustrate in other ways if you wish, though I do think other comments have done so.

 

[Doomtrooper]

Hellbinder, before you go half cocked on me:

Is it true that 2 days after the Doom articles this one came out? Yes or no. Its a fact that can be verified to be correct or not.

I think there was enough hints posted on this forum, and only Beta Members could see it, who is a Futuremark Beta Member.

What does the release date have to with anything ?? Who cares ??
How about telling the truth, which they did..what a concept.

 

[boobs]

c) a bug in 3DM2K3

That only shows up using NVIDIA cards using specific drivers

The instances of software failing on specific vendor platforms are too numerous to list. Nvidia and Ati are pretty much the only players in the DX9 field, saying that it only fails on Nvidia drivers is like saying it only fails 50% of the time, so it must be right.

d) an intentional hack in 3DM2K3

But why?

Nvidia has been dead set against 3DM2K3 since the day it was released. Why wouldn't 3DMark want to discredit a rival? It's working isn't it?

The point is not that 3DMark is doing something unscrupulous, the point, I think, is that at this point there are several alternatives, and there isn't enough evidence to decide what has really happened.

This just goes to show that open source benchmarking needs to be looked at.

 

[WaltC]

Actually, it would seem to me he was stating facts.

2 days after the Doom3 preview, this article making accusations shows up--even though the information has been around for a week or so (apparantly). This is a verifiable to be true or false.
It is heavily rumored that ET was upset they weren't included in the list. This is verifiable to be true or false (though, I suppose what does 'heavily rumored' mean)...

I think you are forgetting that what ET was talking about were the recently released Detonators, a "fact" which you have conveniently omitted. ET said nothing whatever about nVidia's Doom III "demo"--that is entirely Kyle's construct.

So a factual reading of the issue is more like "2 days after the release of nVidia Detonators which nVidia claimed provided a significant performance increase ET ran its Detonator expose'." Also a fact is that ET's findings have been corroborated by B3D and FutureMark. Is it your contention that B3d is also "upset" at not being used crassly by nVidia, and that therefore their corroboration of ET's findings is also suspect? If so, trust me--the sites fortunate enough to have been left out of this morass will come out smelling like a rose. Perhaps, too, you would like to assert as a "fact" that FutureMark is also piling on simply because nVidia resigned from its 3DMark program?

The biggest fact of all, of course, is that neither ET nor B3d nor FutureMark had anything to do with the construction of nVidia's "new, performance enhancing" driver release. nVidia gets all the "credit" there, is the fact of the matter.

People's "polarity" has zero to do with this, Russ. What matters here is evidence, plain and simple. ET and B3D and FutureMark have plenty of that--Kyle and nVidia have none.

 

[Doomtrooper]

Why does all the other cards not show this, do people actually think Futuremark is doing this on purpose ??

I'm sure these members would allow that.

 

[John Reynolds]

I don't think he suggested falsified at all. Ulteriour motive, yes. Falsified? no.

Again, if all you're concerned about are the facts then why be concerned with motives at all? This is just a smear tactic and, IMO, woefully unprofessional. And Kyle's not alone either. I've seen NVNews' webmaster posting a less-than-subtle insinuation that Salvatore didn't even write the article. Both he and Kyle's comments fail to address the means by which the article reaches the conclusion that Nvidia was 'tweaking' their drives to obtain the inflated scores.

What were Kyle's motives for lambasting Matrox like he did last year? I'm sure he wasn't having a childish reaction when he learned that he wouldn't be receiving a Parhelia board to review. Nah, that wasn't it at all. Of course Kyle owns [H] and is free to write whatever springs to mind; Salvatore, as a ZD employee, is not in the same boat.

 

[T2k]

c) a bug in 3DM2K3

That only shows up using NVIDIA cards using specific drivers

The instances of software failing on specific vendor platforms are too numerous to list. Nvidia and Ati are pretty much the only players in the DX9 field, saying that it only fails on Nvidia drivers is like saying it only fails 50% of the time, so it must be right.

d) an intentional hack in 3DM2K3

But why?

Nvidia has been dead set against 3DM2K3 since the day it was released. Why wouldn't 3DMark want to discredit a rival? It's working isn't it?

The point is not that 3DMark is doing something unscrupulous, the point, I think, is that at this point there are several alternatives, and there isn't enough evidence to decide what has really happened.

This just goes to show that open source benchmarking needs to be looked at.

I'm turning back: why NV was strongly against 3dM2k3 from its release? Because they've realized there's no more space left for their fraud?

 

[boobs]

snip

Antlers,

I agree that Nvidia cheating is the possibility that immediately jumps to mind. However, I'm not sure you have the correct assessment of how much work is involved in doing something like that. Furthermore, I have had my initial impressions proved wrong enough times that I would not make a call before more facts are known.

We are pretty much on the same page with regards to benchmarks in general I think.

 

[RussSchultz]

WaltC, for clarification, you're stating that the Extremetech article did not come out 2 days after the Doom3 article posted on HardOCP? You're completely confused about what was stated by Kyle.


And please people, before you string me up as some sort of Kyle effigy. I'm not asserting anything--he is! Though, from my perspective, every fact he has presented is true (and nobody has shown they are not). His opinion is his own, and not mine. I haven't even suggested whether I think NVIDIA is cheating or not. You've lumped me into that category because I'm saying the facts presented are tue.

 

[Clashman]

d) an intentional hack in 3DM2K3

But why?

Nvidia has been dead set against 3DM2K3 since the day it was released. Why wouldn't 3DMark want to discredit a rival? It's working isn't it?

The point is not that 3DMark is doing something unscrupulous, the point, I think, is that at this point there are several alternatives, and there isn't enough evidence to decide what has really happened.

This just goes to show that open source benchmarking needs to be looked at.

Would this even be possible? In order for there to be an intentional hack aimed at discrediting nvidia it seems as though 3dmark would need to detect a specific driver version of nvidia cards, (which it can do), and alter it's engine to screw up the benchmark *only when it is not normally noticeable*. In addition, knowledge that futuremark intentionally attempted to harm the scores of one of the major vendors would likely put them out of business. Is petty anger over nvidia criticism worth risking the fate of your company? Moreover, why mess with this driver set, (or the past two or three, as it may be), instead of one earlier? How could it be based on this driver set at all? As I'm guessing people have had this developer version of 3dmark for some time, they would have to have been planning to *bugger up* this driver set for months before it was even announced.

Am I making sense here?

 

[Natoma]

Correct me if I'm wrong here with regard to the "timing" of the article, but if I'm not mistaken, didn't Rev and DaveBaumann basically state over a week ago that ET was writing an article with regard to "Drivers and 'Optimizations'" thread?

Looking back over what Rev was stating in that thread, and a couple others, it seems that this was something that has been brewing behind the scenes for a while. At least, the information regarding the Nvidia cheating that is. That basically everyone was taking their time before coming out with this story, which imo is a good thing. Vet your information before releasing it to the public.

Considering that ET and B3D seem to be heads and tails technologically superior in terms of knowledge to Kyle/HardOCP, I would tend to believe what they're saying. I mean after all, wasn't it the good people here who found out about the whole 4x2 debacle of the NV30 when Nvidia was saying "We're 8x1 dammit! 8x1 8x1 8x1!!"

Sigh. I hate having to take product reviews with a grain of salt. Talk about lack of journalistic integrity. And I thought the Times had a problem. Now HardOCP eh....