HardOCP's position on the 3DMark2003/Nvidia issue

[Pete]

Towards the end of the huge thread at the [H] forums about Kyle's front-page editorial, the FrgMstr himself posted something that struck me as wrong. He basically said as long as IQ wasn't impacted, then this 3DM03-specific culling is acceptable, as the whole point of 3DM03 is the fastest framerate at the best IQ.

This strikes me as missing the point. IQ isn't the only issue here; rather, the whole purpose of a reproducible and equally-challenging benchmark is. It's not that nVidia's "optimization" doesn't affect IQ (though posts from Ante P and Wavey seem to indicate otherwise), it's that this optimization is contrary to the spirit and point of a general benchmark like 3DM03. 3DM03 is meant to test the speed at which (competing) 3D cards render the exact same scene data with the exact same IQ. (Note that nVidia has now mucked with both qualifiers.) The minute one vendor starts deciding, independently of FutureMark (and thus the other vendors), what portions of the benchmark scene to render, it has effectively invalidated its benchmark scores.

Now, Kyle and nVidia and others can debate separately whether benchmarks that aren't gleaned from an actual game are fair or worthwhile, but that issue is distinct from this. It also holds some merit in the case of benching future features, as I believe there still aren't games on the market that use DX9 features, so time 3D companies spend optimizing for a current benchmark testing future performance is time these companies could perhaps better spend optimizing current games. OTOH, these speculative benchmarks also serve to keep IHV's honest, in that they can verify that the cards they put out do indeed have the features they tout. So there's a trade-off, as always. I'm not sure where I stand on the question of premature optimization.

I just wanted to get this off my chest, as the good big-mouthed forum dweller I am. I stopped reading this thread at around page 5, so my apologies if this has already been belabored.

Edit: Unbelievable, another post that ends up at the top of a new page. In this case I'm actually pleased, as I think this post is more worthwhile than mine usually are.

 

[stevem]

why in the world would you buy something that is being misrepresented?

Ask AV enthusiasts. They do it on a regular basis.

Damn! And I just bought matched, directional 8AWG silver crystal speaker cables with an amp low C stabilizer... What's a Zobel...?

 

[Volenti]

Damn! And I just bought matched, directional 8AWG silver crystal speaker cables with an amp low C stabilizer... What's a Zobel...?

a Zobel does something usefull at least

 

[Doomtrooper]

Problem is there is 'image degradation, and not just 3Dmark but Serious Sam 2...he needs his head examined...and double standards thrown out the window.

They just don't seem to use logic, if a IHV Optimizes UT 2003 flyby using the same technique, they feel thats ok (even though the Principle of optimization is the same as 3Dmark= higher score)..and image quality issues.

 

[WaltC]

Towards the end of the huge thread at the [H] forums about Kyle's front-page editorial, the FrgMstr himself posted something that struck me as wrong. He basically said as long as IQ wasn't impacted, then this 3DM03-specific culling is acceptable, as the whole point of 3DM03 is the fastest framerate at the best IQ.
...

Then Kyle has completely missed the point, so zealous is he in defending nVidia's right to cheat.

He must consider himself an expert at divining the motives of others, apparently, so how is it he completely misses nVidia's motive for cheating the benchmark?

nVidia cheated it in order to raise the benchmark scores above what they would be if nVidia's drivers were rendering 3D Mark like, say, ATi's drivers render it. Despite its publicly expressed disdain for 3DMark 03, nVidia must see 3DMark 03 benchmark scores as an important marketing tool relative to the sale of its products, else the company would never have taken the time to learn to cheat the benchmark, and then to release driver sets which actually do cheat it.

Why it matters, and why Kyle is wrong, is because the comparisons of nVidia's products relative to ATI's products (or anyone else's) become completely invalid as a result of the cheating, since ATi's drivers render normally, and thus the ATi vpu is doing a lot more work while running the benchmark than is nVidia's gpu. Hence any comparisons with ATi products running 3DMark 03, or anyone else's products which do not cheat the benchmark, is no longer possible.

The only way for Kyle's suggestion to make any logical sense whatsoever is if it was not possible to run any other hardware apart from nVidia's with the 3DMark 03 software. Then and only then would this kind of cheating not matter. But since the purpose of the benchmark is to allow end users to contrast and compare vpus as to performance, and since nVidia's driver cheat makes that impossible since it is doing less work than contrasting chips running drivers which do not cheat in the same way, comparing nVidia products to others through the vehicle of 3D Mark then becomes an impossibility.

 

[Clashman]

Edit: Unbelievable, another post that ends up at the top of a new page. In this case I'm actually pleased, as I think this post is more worthwhile than mine usually are.

What news page?

 

[Xmas]

Edit: Unbelievable, another post that ends up at the top of a new page. In this case I'm actually pleased, as I think this post is more worthwhile than mine usually are.

What news page?

A new page (in this thread), not a news page.

 

[Brent]

Problem is there is 'image degradation, and not just 3Dmark but Serious Sam 2...he needs his head examined...and double standards thrown out the window.

They just don't seem to use logic, if a IHV Optimizes UT 2003 flyby using the same technique, they feel thats ok (even though the Principle of optimization is the same as 3Dmark= higher score)..and image quality issues.

I don't think its ok

In a gaming benchmark such as UT2K3 the whole idea is for that flyby to represent actual gameplay performance. If an IHV optimizes that benchmark demo by clipping away other textures and so forth, then that demo no longer represents what you actually experience in the game.

Optimizing is increasing performance or effeciency without degrading image quality and gameplay experience. So in the example above that IHV would be cheating.

I wouldn't tolerate that either

 

[Tim]

I don't think its ok

In a gaming benchmark such as UT2K3 the whole idea is for that flyby to represent actual gameplay performance.

And whole idea with 3dmark is to simulate actual games I completely fail to see the difference.

And it has absolutely no relevance if 3Dmark is doing a good job at this or not.

If an IHV optimizes that benchmark demo by clipping away other textures and so forth, then that demo no longer represents what you actually experience in the game.

And the same holds true with 3dmark, the scores no longer represents the performance in hypothetical Troll Lair game or Mother Nature game.

Optimizing is increasing performance or effeciency without degrading image quality and gameplay experience. So in the example above that IHV would be cheating.

I wouldn't tolerate that either

I still fail to understand why it is ok to do it a benchmark that simulates a timedemo, but not ok in an actual timedemo.

 

[boobs]

I don't think its ok

In a gaming benchmark such as UT2K3 the whole idea is for that flyby to represent actual gameplay performance.

And whole idea with 3dmark is to simulate actual games I completely fail to see the difference.

And it has absolutely no relevance if 3Dmark is doing a good job at this or not.

If an IHV optimizes that benchmark demo by clipping away other textures and so forth, then that demo no longer represents what you actually experience in the game.

And the same holds true with 3dmark, the scores no longer represents the performance in hypothetical Troll Lair game or Mother Nature game.

Optimizing is increasing performance or effeciency without degrading image quality and gameplay experience. So in the example above that IHV would be cheating.

I wouldn't tolerate that either

I still fail to understand why it is ok to do it a benchmark that simulates a timedemo, but not ok in an actual timedemo.

I think what Brent means can be illustrated by the following example.

Imagine if a consumer looked at benchmark-X scores to make his purchasing decision between two hypothetical cards.

Card A uses every available transistor to maximize pixel shading performance.

Card B uses a balance of pixel shading and occlusion rejection, etc.

Since benchmark-x is coded in a way that makes occlusion rejection largely irrelevant, card A wins the consumer over, when, in reality, card B would have provided better performance in the games that the consumer would want to play. In this situation, the consumer has lost out.

If this happens on a wide scale, companies might be pressured by the market to start making future chips as unbalanced as that found in card A rather than card B. In that case, consumers in general have lost out.

To prevent that from happening, company B puts in custom "optimizations" that simulate the effects of occlusion rejection to make performance in benchmark-X more akin to what one would find in real games. As a result, the consumer chooses card B, thus gaining benefit from "cheats" by company B.

Conclusions from this situation?

Company B are scoundrels.
The consumer may actually benefit from company B being scoundrels.
Benchmark-X is clearly worthless as a benchmark.

What is the best way to move forward in a way that benefits the consumer?

Use real applications as benchmarks.
Takes steps to prevent cheating.
Educate the consumer about using a variety of sources and real applications to evaluate their purchasing decision.

[H] is doing the above, and deserves applause.

 

[WaltC]

And whole idea with 3dmark is to simulate actual games I completely fail to see the difference.

And it has absolutely no relevance if 3Dmark is doing a good job at this or not.
...

I think the whole idea behind 3DMark 03 is to test various areas of DX9-compliant vpus for performance specifically relating to DX9 features (I think 95% or or so of the bench centers around the DX9 feature set.)

The purpose of the bench is not to simulate games but to compare and contrast vpus on a level, DX9-centered playing field, IMO. If one manufacturer's driver cheats the bench by doing less work while running the bench than another manufacturer's driver, then the bench is no longer valid as a comparison since there is a difference in the workload each vpu is doing while running it. The same principle would apply to a fly-by demo, a time demo, or any other comparative performance-ranking tool which features a camera on a track. Unless the workload undertaken by the compared vpus is identical, the fps results of running the software cannot be used in a performance comparison.

 

[madshi]

Brent, will you be using the Code Creatures benchmark in the future?

 

[Randell]

Imagine if a consumer looked at benchmark-X scores to make his purchasing decision between two hypothetical cards.

Card A uses every available transistor to maximize pixel shading performance.

Card B uses a balance of pixel shading and occlusion rejection, etc.

Since benchmark-x is coded in a way that makes occlusion rejection largely irrelevant, card A wins the consumer over, when, in reality, card B would have provided better performance in the games that the consumer would want to play. In this situation, the consumer has lost out.

If this happens on a wide scale, companies might be pressured by the market to start making future chips as unbalanced as that found in card A rather than card B. In that case, consumers in general have lost out.

To prevent that from happening, company B puts in custom "optimizations" that simulate the effects of occlusion rejection to make performance in benchmark-X more akin to what one would find in real games. As a result, the consumer chooses card B, thus gaining benefit from "cheats" by company B.

Conclusions from this situation?

Company B are scoundrels.
The consumer may actually benefit from company B being scoundrels.
Benchmark-X is clearly worthless as a benchmark.

What is the best way to move forward in a way that benefits the consumer?

Use real applications as benchmarks.
Takes steps to prevent cheating.
Educate the consumer about using a variety of sources and real applications to evaluate their purchasing decision.

[H] is doing the above, and deserves applause.

Dont try to make this nVidia is benefiting the consumer by being cheating scoundrels. Game cameras dont work on rails - occluding where the camera is known not to be lookng isnt possible in a 3d game. The above post is just BS and I'd be pretty surprised if Brent agrees with what you dsay he is saying.

[H] do not deserve applause, the opening sentence from Kyle in his editorial is gutter journalism - not investigative journalism.

 

[Tim]

I think what Brent means can be illustrated by the following example.

Imagine if a consumer looked at benchmark-X scores to make his purchasing decision between two hypothetical cards.

Card A uses every available transistor to maximize pixel shading performance.

Card B uses a balance of pixel shading and occlusion rejection, etc.

You are aware that the R350 are doing better than the NV3x when it comes to occlusion culling etc?

Even if you were not everything you wrote was so far out, that we actually got dumber just by reading it.

 

[boobs]

Dont try to make this nVidia is benefiting the consumer by being cheating scoundrels. Game cameras dont work on rails - occluding where the camera is known not to be lookng isnt possible in a 3d game. The above post is just BS and I'd be pretty surprised if Brent agrees with what you dsay he is saying.

[H] do not deserve applause, the opening sentence from Kyle in his editorial is gutter journalism - not investigative journalism.

The point is that no self respecting game developer would have done the demo the way that 3dMark did, yet, that demo is being represented to the consumer as a benchmark for future games.

Gutter journalism is putting up misrepresentations, not speaking your mind in an editorial.

 

[jb]

Brent,

I gotta disagree with that. The bases on weather a certian IHV cheats/bugs/optimizes their drivers does not make the benchmarks any more/less valid. ATI showed that you could "opti-cheat?" Q3 so by your own means you might as well stop using Q3? You know that many buying decsisions are based of 3dmarks scores be that right or wrong its just a fact. Scores that are inflated then do everyone a dis-service.

 

[WaltC]

....
Conclusions from this situation?

Company B are scoundrels.
The consumer may actually benefit from company B being scoundrels.
Benchmark-X is clearly worthless as a benchmark....

A very odd and convoluted opinion, Boobs...

If we call company B a "scoundrel" because it misrepresents the amount of work its vpu is doing in a given amount of time, to the degree that the consumer reaches an erroneous opinion about how much work the vpu can do in that amount of time, contrasted with competing vpus, how is this in the consumer's interests?

I cannot possibly see how. What the consumer will discover is that when his vpu is operating in software the performance of which cannot be misrepresented by the driver in the same way, that his performance will be lower than he expects. Certainly lower than he expected based on the erroneous results provided by the benchmark--which result not from the benchmark itself--but from company B's drivers which cheat it.

Therefore, company B is doing a major number on the consumer, and it is company B's drivers which are "worthless"--instead of the benchmark, which played no part in the deception. Hence company B is indeed a scoundrel.

[H]'s assertions in this matter have no basis in either logic or fact.

 

[boobs]

You are aware that the R350 are doing better than the NV3x when it comes to occlusion culling etc?

I'm talking about the need for a benchmark to assess performance in a realistic manner, and giving that as a hypothetical example. Which specific card is faster is irrelevant to my point.

Even if you were not everything you wrote was so far out, that we actually got dumber just by reading it.

It must be pretty difficult for you to get dumber huh Tim?

 

[Tim]

I think the whole idea behind 3DMark 03 is to test various areas of DX9-compliant vpus for performance specifically relating to DX9 features (I think 95% or or so of the bench centers around the DX9 feature set.)

The 3dmark score are based on four "game" tests one DX7, two DX8 and one DX) test, the DX9 game test only accounts for around 20% of the total score.

The purpose of the bench is not to simulate games but to compare and contrast vpus on a level, DX9-centered playing field, IMO.

The high quality game tests, image quality tests, sound tests and others give you an extremely accurate overview of your system’s current gaming performance.

I think it is clear that simulating current and future game performance is one of the goals of 3dmark.

 

[Randell]

The point is that no self respecting game developer would have done the demo the way that 3dMark did, yet, that demo is being represented to the consumer as a benchmark for future games.

Gutter journalism is putting up misrepresentations, not speaking your mind in an editorial.

two points.

1. No matter how you twist it, the pros and cons of 3dMark as a benchmark does not justify cheating in it in the manner presented.

2. Kyle presented his opinion that ET's motivation was revenge - which is a misrepresentation as DS had been working on this for a while, as verfied by Wavey and Rev.