HardOCP's position on the 3DMark2003/Nvidia issue

[Entropy]

Yeah, but they still buy the products.

Again - speak for yourself.

Entropy

 

[boobs]

Again - speak for yourself.

Entropy

Is it not a fact that consumers buy products from these companies?

 

[OpenGL guy]

That is true for today's hardware but I believe many older chips are faster without a separate Z clear.

It depends, but I doubt it makes much difference because the bandwidth consumed is the same either way, assuming a lack of Z optimizations.

It's almost guaranteed to be true, as the only difference is that only visible sky pixels are rendered. How could anyone come up with such a ridiculous idea like rendering the sky first?
So, yes, I would also say it's the wrong order.

If you're talking about game performance, then it makes a difference because you want the game to go as fast as possible. If you're talking about giving the card a certain workload then it doesn't matter if the sky comes first. I don't speak for FutureMark, but the goal of 3D Mark 2003 is not to get 100 fps but to create a workload for your video card. If your card is slow at this workload, tough luck.

 

[digitalwanderer]

Towards the end of the huge thread at the [H] forums about Kyle's front-page editorial, the FrgMstr himself posted something that struck me as wrong. He basically said as long as IQ wasn't impacted, then this 3DM03-specific culling is acceptable, as the whole point of 3DM03 is the fastest framerate at the best IQ.
...

Then Kyle has completely missed the point, so zealous is he in defending nVidia's right to cheat.

He must consider himself an expert at divining the motives of others, apparently, so how is it he completely misses nVidia's motive for cheating the benchmark?

nVidia cheated it in order to raise the benchmark scores above what they would be if nVidia's drivers were rendering 3D Mark like, say, ATi's drivers render it. Despite its publicly expressed disdain for 3DMark 03, nVidia must see 3DMark 03 benchmark scores as an important marketing tool relative to the sale of its products, else the company would never have taken the time to learn to cheat the benchmark, and then to release driver sets which actually do cheat it.

Why it matters, and why Kyle is wrong, is because the comparisons of nVidia's products relative to ATI's products (or anyone else's) become completely invalid as a result of the cheating, since ATi's drivers render normally, and thus the ATi vpu is doing a lot more work while running the benchmark than is nVidia's gpu. Hence any comparisons with ATi products running 3DMark 03, or anyone else's products which do not cheat the benchmark, is no longer possible.

The only way for Kyle's suggestion to make any logical sense whatsoever is if it was not possible to run any other hardware apart from nVidia's with the 3DMark 03 software. Then and only then would this kind of cheating not matter. But since the purpose of the benchmark is to allow end users to contrast and compare vpus as to performance, and since nVidia's driver cheat makes that impossible since it is doing less work than contrasting chips running drivers which do not cheat in the same way, comparing nVidia products to others through the vehicle of 3D Mark then becomes an impossibility.

Will you be me new hero Walt? I'm loving all your posts in this thread!!!!

I was arguing this one with Kyle yesterday and it came down to this sticking point. He didn't think there was anything wrong with them not rendering the unseen bits since it didn't impact the visuals and he just didn't see a problem with it.

I SEE A PROBLEM WITH IT! You summed it up nicely, where were ya yesterday when I needed you?

 

[digitalwanderer]

I don't think its ok

In a gaming benchmark such as UT2K3 the whole idea is for that flyby to represent actual gameplay performance. If an IHV optimizes that benchmark demo by clipping away other textures and so forth, then that demo no longer represents what you actually experience in the game.

Optimizing is increasing performance or effeciency without degrading image quality and gameplay experience. So in the example above that IHV would be cheating.

I wouldn't tolerate that either

I'm glad to hear you don't support it Brent, I really am.

Then why don't you speak out against the 3dm2k3 cheat? Please don't tell me you're still waiting around to hear nVidia's "official BS"....it don't take that long to tell the truth!

Is everyone at [H] in agreement over this one? That was the impression I got yesterday from Kyle, but it was only an impression.

 

[stevem]

Damn! And I just bought matched, directional 8AWG silver crystal speaker cables with an amp low C stabilizer... What's a Zobel...?

a Zobel does something usefull at least

In a Xover, no doubt. However, I don't need one at the speaker terminals of my amp to compensate for a poor output stage when using poorly designed, highly capacitative cables... HK, anyone...?

 

[WaltC]

Will you be me new hero Walt? I'm loving all your posts in this thread!!!!

I was arguing this one with Kyle yesterday and it came down to this sticking point. He didn't think there was anything wrong with them not rendering the unseen bits since it didn't impact the visuals and he just didn't see a problem with it.

I SEE A PROBLEM WITH IT! You summed it up nicely, where were ya yesterday when I needed you?

Hardly "hero" material, I'm afraid... (Although a "god on earth" has a nice ring to it....*chuckle*)....

Seriously, if by this time Kyle doesn't understand why it isn't cricket to compare a vpu doing X amount of work with a vpu doing an X-1x amount of work, I guess he's never going to. I'll wager the driver guys at nVidia have no trouble whatever with the concept, however...

 

[nelg]

Hey, digitalwanderer when was the last time you slept. You are everyware.

 

[Entropy]

Is it not a fact that consumers buy products from these companies?

Of course.
But a certain percentage of the customers may realize that one part of the price/performance equation is not directly under their control, but that they rely on test-data supplied to them. If they realize that they have been systematically decieved by an IHV, how (and why) could they have any faith in such data in the future?

Customers that might take a dim view on being systematically and consistently decieved could include, for instance, Dell.

Goodwill and trust are valuable commodities. As in dollars and cents.

Why financially support a particularly dishonest hardware manufacturer?

Entropy

 

[sabeehali]

I don't think its ok

It is a cheat (based on the Extremetech analysis), and here is why....and most people aren't looking at it in the correct way:

If the basis of your optimization requires to to have access to data that is NOT PASSED by the game engine in real time, then that optimization is a cheat. This 3DMark cheat is based on the fact that the drivers "are told" the camera path won't change from some determined path. Problem is, they are not told this by the game engine. Clipping planes are inserted based on this knowledge. That data (the clipping planes) are not passed from the engine in real-time, nor are those planes calculated in real-time (as evidenced by the lack of correct rendering when "off the rail".)

That is why this is particular exanple is a cheat, and not a legal optimization. It relies on data that is not given by the benchmark, or calculated in real-time from data given by the benchmark.

This is why something like a "deferred renderer" is NOT cheating. It's not drawing "everything" either. But it calculates, on the fly, frame by frame, what is needed to be drawn. If you took a deferred renderer "off the rail" it would not suffer the clipping issues.

Thank YOU

I was hoping after 10 pages somebody would point it out.

 

[Deflection]

You know what they say about the difference between a good thief and a bad thief.

So I'm the bad guy now, eh? If it makes you feel better to think so, then go right ahead; I'm not going to lose any sleep over it.

Please don't. Perhaps that was a poor choice of words. I would like to apologize for that, as I certainly am not trying to impugn your integrity personally. The gist of what I was trying to get across in my posts in this thread is that:

- all 3D companies do game/benchmark specific optimizations and will continue to do so.
- often the line between optimization and "cheating" can be very thin. Especially in a highly competitve industry whose goal is to intelligently render high quality 3D as fast as possible. This involves a lot of cost/benefit tradeoffs.
- I prefer to not single out any 3D company for their morality or lack thereof as I do not think many companies in general qualify for sainthood. I am not against pointing out Nvidia's cheat in 3DMark. I applaud it. However, I don't think people should get too carried away with it in light of the history of the industry.
- Kyle statement perhaps unfairly characterizes ET. ET's statement that Nvidia cheated perhaps unfairly characterizes Nvidia. In the end, this is the internet and people will write what they want. Websites and messageboard posters will continue to feed on rumor and innuendo. Thus my "So what?".

Keep bringing us great 3D graphics at ATI

 

[madshi]

In the end, this is the internet and people will write what they want. Websites and messageboard posters will continue to feed on rumor and innuendo. Thus my "So what?".

So you accept the status quo as being given? This way you almost legitimate websites throwing dirt at each other and IHVs cheating their life out of them. Why should they change their behaviour if people think like you seemingly do? I'm just glad that lots of people are willing to stand up and complain, as can be seen in this thread.

 

[jjayb]

I find it kind of odd that of the 3 websites that got to bench doom3 with Nvidia, only one bothered to mention the ET article. Yet the one that did mention it(hardocp of course) tried to defend Nvidia by blasting ET and saying ET was just mad because they didn't get to bench doom3. He then went on to defend Nvidia heavily. You wouldn't even know the article existed by looking at the other 2 sites. Hmm, makes you wonder.

Hey if Kyle can speculate I can too.

 

[no_way]

Open source The Benchmark.
So the benchmark code can be patched, updated, bullet-proofed and customized faster than IHVs driver code.

Code wars!

 

[demalion]

Open source The Benchmark.
So the benchmark code can be patched, updated, bullet-proofed and customized faster than IHVs driver code.

Code wars!

You do realize that IHVs would have access to the source code too, right?

 

[no_way]

You do realize that IHVs would have access to the source code too, right?

Absolutely. But what harm can come of it ? Yes, they can identify coding patterns and try to match them in their drivers. Better outcome would be, that they openly state that one or other rendering method is not "legal" or hampers performance. But other IHVs, plus people who actually write real game engines could give their input too, so the what exactly is the "Right Way" would be up to all interested parties.
Even if "driver hacks" could be introduced, such hacks would be very easily exposed as everyone, including review sites would have access to benchmark code.
Plus, anyone who whines about how the benchmark is not proper representative of real-world games, can take it on and improve it.

 

[Ilfirin]

Yeah.. like a review site is going to weed through tens (or hundreds) of thousands of lines of code to see if a card is hacking its way through the bench..

And what would that do anyway? The problem isn't in the bench's code, it's in the driver's.


Getting more on topic - If nVidia can detect that a predictable benchmark is being run and turn on hacks, they can just as easily detect when the camera is going in areas the driver didn't expect, and turn off the "optimizations"; making it nearly impossible to detect the hacks (sure you might see a slight framerate drop when it happens, but that could just as easily be written off as a change in scene complexity or a slight framerate dip due to a background process.. it probably wouldn't be any more than 2-5 fps). Maybe they had that functionality in the drivers and it didn't work, hence nVidia calling it a 'bug'?

 

[YeuEmMaiMai]

LOL bug my butt.....they got caught cheating plain and simple

 

[Ilfirin]

You obviously didn't get what I said

 

[WaltC]

- all 3D companies do game/benchmark specific optimizations and will continue to do so.

OK.

- often the line between optimization and "cheating" can be very thin. Especially in a highly competitve industry whose goal is to intelligently render high quality 3D as fast as possible. This involves a lot of cost/benefit tradeoffs.

As you point out there is a definite line between cheating and optimizing. They are not the same thing, is the important thing to remember. Other than attempting to defend a company caught cheating, what is the purpose of blurring the line between optimizing and cheating?

- I prefer to not single out any 3D company for their morality or lack thereof as I do not think many companies in general qualify for sainthood. I am not against pointing out Nvidia's cheat in 3DMark. I applaud it. However, I don't think people should get too carried away with it in light of the history of the industry.

Which do you applaud--the cheating or its exposure? *chuckle* (I know what you meant to say--just kidding.)

Well, let's extend your philosophy to other areas. If I get defrauded on eBay because of an unscrupulous person who cheats me, am I "carried away" if I seek a remedy and to expose the cheat? In other words, in any venue in life which has to do with companies or individuals making false claims about their products is it "improper" to get "carried away" simply because dishonesty occurs elsewhere? More specifically, does a company have the "right" to cheat now simply because it (or other companies) has cheated in the past?

While the history of the industry certainly includes past examples of cheating, it also includes numerous examples of companies who have been caught cheating and suffered because of it. Is there any reason to expect that a company should not suffer if it is caught cheating?

Saying something like "Who cares if nVidia is cheating presently because in the past ATi cheated, too," strikes me as rather like saying, "It's no big deal if MCI rips me off on my telephone bill this month--because AT&T ripped me off last year." It's not a profitable train of reasoning...

- Kyle statement perhaps unfairly characterizes ET. ET's statement that Nvidia cheated perhaps unfairly characterizes Nvidia. In the end, this is the internet and people will write what they want. Websites and messageboard posters will continue to feed on rumor and innuendo. Thus my "So what?".

Keep bringing us great 3D graphics at ATI

Actually, what ET did was to present the facts as they discovered them, along with nVidia's "driver bug" explanation, and leave judgment up to the reader. The same facts have been confirmed by FutureMark and B3d.

What Kyle did was seek to impugn ET's motivation for disclosing those facts (with lame comments about his telepathic intuition that ET was upset about being passed over for the whoop-te-doo, made by-and-for-nVidia Doom III demo--as the great majority of web sites were likewise passed over for it.) Unlike the ET presentation, Kyle presented not one--not a single one--relevant fact to back up his telepathic impressions. He did not even attempt to disprove the data as ET presented it, and has subsequently not attempted to disprove it.

ET made a case; [H] did not. Paraphrased, Kyle's postion has been, "If you can't SEE the cheating, then it's not cheating--it's optimization." Kyle's central problem is that when the camera goes off track, you can then plainly SEE the cheating... One can only think that either Kyle simply does not understand the issues, or else he does not wish to understand them. Ironically, Kyle's bizarre repudiation of ET's motives has boadened the profile of this issue somewhat over what it might have been otherwise.

The only possible inference that can be gathered as to what it was Kyle would have felt was acceptable behavior for ET is that Kyle feels ET should have covered up what it found as opposed to exposing it and asking nVidia for an explanation.

I don't think you will benefit much by equating the case ET made with Kyle's orgasmic attempts to assassinate the character of those behind the ET website over this matter. ET's facts can be repeated and verified and this has already occurred; Kyle's comments have been repeated and verified by no one. I think that's an important distinction.