HardOCP's position on the 3DMark2003/Nvidia issue

[boobs]

A very odd and convoluted opinion, Boobs...

If we call company B a "scoundrel" because it misrepresents the amount of work its vpu is doing in a given amount of time, to the degree that the consumer reaches an erroneous opinion about how much work the vpu can do in that amount of time, contrasted with competing vpus, how is this in the consumer's interests?

I cannot possibly see how. What the consumer will discover is that when his vpu is operating in software the performance of which cannot be misrepresented by the driver in the same way, that his performance will be lower than he expects. Certainly lower than he expected based on the erroneous results provided by the benchmark--which result not from the benchmark itself--but from company B's drivers which cheat it.

Therefore, company B is doing a major number on the consumer, and it is company B's drivers which are "worthless"--instead of the benchmark, which played no part in the deception. Hence company B is indeed a scoundrel.

[H]'s assertions in this matter have no basis in either logic or fact.

That not a rigorous analysis.

Company B is a scoundrel because it cheated on the benchmark out of its selfish interest.

The benchmark is worthless because it does not represent realistic applications.

Since the benchmark is worthless, it may actually turn out that company B's cheats will lead to consumer to a better purchasing decision.

Convoluted? Yes! Wrong? No!

Does the consumer really care that company B is a scoundrel? Not really, the consumer just wants to buy a good video card, just like the way the consumer continues to buy MS software, use Worldcom for phone service, pump gass from Chevron, buy cookies from Philip Morris, etc.

 

[WaltC]

The 3dmark score are based on four "game" tests one DX7, two DX8 and one DX) test, the DX9 game test only accounts for around 20% of the total score.

I won't argue that with you at the moment, but I will ask you to consider why it is that nVidia's objection to the benchmark is that it doesn't accurately represent the way 3D games will be written in the future. At least, that's what nVidia gave as the reason it pulled out of the program. If it is not a future-based bench, just for the sake of argument, then you have described it as a backwards-looking program. So what is nVidia's objection to it based on?

I think it is clear that simulating current and future game performance is one of the goals of 3dmark.

Actually, "simulating games" is something 3DMark does, apart from a "goal." FutureMark has made it plain that '03 is a vpu benchmark and not the type of system benchmark that '02 certainly was. The goal of '03 is to primarily guage vpu performance, at this stage no one but nVidia sems to have a problem with it. If that problem nVidia has is not '03's approach to DX9, then what on earth could it be?

 

[Randell]

Since the benchmark is worthless, it may actually turn out that company B's cheats will lead to consumer to a better purchasing decision.

if you havent already got one I suggest you seek a job in nVidia's PR department

 

[indio]

I really don't get this stuff about 3dmark. 3dmark is not a game . It is a tool to determine 3d performance relative to other cards. Please enlighten me and show me where 3dmark consistantly gets this wrong?
Has any ever got a 3dmark score that showed a GF4MX beating a GF4-4600? I suspect the anwser is no. Saying 3dmark is invalid becuase i doesn't represent an actual game is bunk. If it did represent an actual game ppl. would say that game is coded this way or that and is not representitive of some other game.
Lets' try an analogy.
I build a custom 2 mile track with all sorts of ups and downs and left handed turns etc.. then I drive Jeff Gordons car on it and it beats my Nissan. After that someone says well Nascar tracks are all ovals so the test is inaccurate cause tracks aren't built like that. Does that mean my Nissan is faster than Jeff Gordan's car>?
Of course not. All I see is people saying the test is worthless and have yet to see any evidence where it gets it wrong. If you wanna see how your card does in Q3 , test in Q3. If you want to know how your card will perform generally across all games use 3dmark.

 

[boobs]

Since the benchmark is worthless, it may actually turn out that company B's cheats will lead to consumer to a better purchasing decision.

if you havent already got one I suggest you seek a job in nVidia's PR department

LOL, what would they want with a physicist/material scientist?

I'd prefer being a lawyer over a PR rep in any case.

I don't see why anyone would passionately attack one company over another. Malfeascence abound in the corporate world, it's human nature.

All I really care about is getting a good video card to play DOOM III. That and having fun poking people who put moral credence into video game benchmarketing.

 

[Joe DeFuria]

I don't think its ok

In a gaming benchmark such as UT2K3 the whole idea is for that flyby to represent actual gameplay performance.

First of all, you're wrong in the premise of UT2K3 demos.

There are two types of UT2K3 demos: flybys and botmatches. They are both designed to do different things. "Flybys" do NOT represent actual gameplay performance. In fact, it's quite the opposite. The FlyBy demos are meant to NOT represent gameplay performance, by forcing GPU limitations to a far greater extent than actual gameplay. (Which bot-matches are suppoed to more or less represent.)

The "purpose" of the UT2K3 "Flyby" type demos are more or less EXACTLY like the purpose of 3DMark. To stress the GPU. FULL STOP. The main difference is:

1) UT is really a DX7 engine, not a DX8/DX9 engine, so UT will stress the GPUs differently than 3DMark.
2) The UT test uses an actual game engine, and 3D Mark doesn't.

But again, I stress that UT Flyby demos are very much the same as 3DMArk 03 in the sense that they do not represent "actual gameplay performance." They try and represent GPU performance.

You can no more predict "actual gameplay performance" of UT or any other DX7 game with a UT2K3 flyby demo, than you can predict actual gameplay DX8/9 gameplay performance with 3DMark.

So what good are they?

As we know, sometimes actual gameplay performace is not particularly GPU limited. Running a bunch of benchmarks that have heavy reliance on CPU or other non GPU related things, doesn't tell us much about the performance differences between GPUs.

That's exactly why there have always been "BotMatch" and "FlyBy" demos for UT. Both are useful...they just tell us different things.

Optimizing is increasing performance or effeciency without degrading image quality and gameplay experience.

That definition is incomplete. This definition of "optimizing" would indicate that the 3DMark clipping is an optimization, not a cheat. It is a cheat (based on the Extremetech analysis), and here is why....and most people aren't looking at it in the correct way:

If the basis of your optimization requires to to have access to data that is NOT PASSED by the game engine in real time, then that optimization is a cheat. This 3DMark cheat is based on the fact that the drivers "are told" the camera path won't change from some determined path. Problem is, they are not told this by the game engine. Clipping planes are inserted based on this knowledge. That data (the clipping planes) are not passed from the engine in real-time, nor are those planes calculated in real-time (as evidenced by the lack of correct rendering when "off the rail".)

That is why this is particular exanple is a cheat, and not a legal optimization. It relies on data that is not given by the benchmark, or calculated in real-time from data given by the benchmark.

This is why something like a "deferred renderer" is NOT cheating. It's not drawing "everything" either. But it calculates, on the fly, frame by frame, what is needed to be drawn. If you took a deferred renderer "off the rail" it would not suffer the clipping issues.

 

[breez]

boobs, your hypothetical situation about 'company b and benchmark x' is just hypothetical. There's no real world situation like it going on...

 

[nelg]

I really don't get this stuff about 3dmark. 3dmark is not a game . It is a tool to determine 3d performance relative to other cards. Please enlighten me and show me where 3dmark consistantly gets this wrong?
Has any ever got a 3dmark score that showed a GF4MX beating a GF4-4600? I suspect the anwser is no. Saying 3dmark is invalid becuase i doesn't represent an actual game is bunk. If it did represent an actual game ppl. would say that game is coded this way or that and is not representitive of some other game.

Well said. With respect to other programs used to test various aspects of video cards (shader performance, AA, etc.) They are not games. Are you, boobs, suggesting that they provide no relevant info. inregards to performance ?

 

[WaltC]

That not a rigorous analysis.

Company B is a scoundrel because it cheated on the benchmark out of its selfish interest.

Right. Which was to misrepresent the performance of its products in order to influence sales.

The benchmark is worthless because it does not represent realistic applications.

This is where you aren't coming in clearly. The benchmark is a benchmark in this case and is not a real-world application. This is true of all benchmarks. 3D Mark is not unusual or an exception in that regard.

If your suggestion is to throw out all benchmarks, all time demos, and all commercial demos (such as nVidia's recent Doom III demo), I can't say I would disagree with that. However, I think you'll agree with me that this is not likely to happen... There is nothing in 3D Mark 03 to set it apart from any of these other kinds of benchmarks. Why is it OK for nVidia to pay for its own Doom III demos and foist those on an unsuspecting public, but it's not OK to run 3D Mark 03 because it is "worthless"...? That certainly seems to be [H]'s position in a nutshell.

Therefore, if 3D Mark 03 is "worthless" than so are all of them "worthless," including nVidia's recent Doom III demo. Or do you think the only benchmarks that should be thrown out are those nVidia deems worthless...?....

Since the benchmark is worthless, it may actually turn out that company B's cheats will lead to consumer to a better purchasing decision.

How so--if the consumer is mislead into buying an underperforming product?

The point you are missing is that what nVidia has done here is misrepresent a product--throwing out 3D Mark or not has absolutely nothing to do with it, since the people who like 3D Mark aren't going to quit using it because nVidia's drivers cheat it. Rather, they will be tempted to throw out nVidia, instead...

Does the consumer really care that company B is a scoundrel? Not really...

I think you can be assured you are pretty much speaking for yourself here, as most "consumers" take a dim view of dealing with scoundrel corporations who brazenly insult their intelligence and attempt to rip them off at the same time....

 

[indio]

we need the Mother of All Benchmarks(TM) marks! it will run every game since 1995 at every resolution 3 times on a random demo path. Then it will be representive of real games (screw you pre-95 !) I just hope I can get it installed before dx10 comes out!

 

[boobs]

I think you can be assured you are pretty much speaking for yourself here, as most "consumers" take a dim view of dealing with scoundrel corporations who brazenly insult their intelligence and attempt to rip them off at the same time....

Yeah, but they still buy the products.

No benchmark can match real life performance, but they should make every effort to do so!

Something like rendering the scene in the wrong order, which is what that particular benchmark did, pretty much make it worthless.

Worthlessness is in degrees, maybe I should follow the fad and put a worthlessness meter in my sig.

 

[OpenGL guy]

No benchmark can match real life performance, but they should make every effort to do so!

Something like rendering the scene in the wrong order, which is what that particular benchmark did, pretty much make it worthless.

In the wrong order? You mean sky first? Do you realize that nearly every game does this?

 

[Brent]

I don't think its ok

In a gaming benchmark such as UT2K3 the whole idea is for that flyby to represent actual gameplay performance.

And whole idea with 3dmark is to simulate actual games I completely fail to see the difference.

And it has absolutely no relevance if 3Dmark is doing a good job at this or not.

If an IHV optimizes that benchmark demo by clipping away other textures and so forth, then that demo no longer represents what you actually experience in the game.

And the same holds true with 3dmark, the scores no longer represents the performance in hypothetical Troll Lair game or Mother Nature game.

Optimizing is increasing performance or effeciency without degrading image quality and gameplay experience. So in the example above that IHV would be cheating.

I wouldn't tolerate that either

I still fail to understand why it is ok to do it a benchmark that simulates a timedemo, but not ok in an actual timedemo.

You could apply what i'm saying to 3DMark03

but there is one main difference that I see, I was refering to a real game. 3DMark03 is not a game. It is synthetic scripted tests using no known game engine, you can't pick it up and run around in Game Test 4 in a first person shooter environment multiplaying with other people (though that would be kinda fun )

3dmark isn't sold as game, its sold as a benchmark

but i agree cheating in benchmarks shouldn't happen either, cheating is cheating no matter what the application

 

[Brent]

And whole idea with 3dmark is to simulate actual games I completely fail to see the difference.

And it has absolutely no relevance if 3Dmark is doing a good job at this or not.
...

I think the whole idea behind 3DMark 03 is to test various areas of DX9-compliant vpus for performance specifically relating to DX9 features (I think 95% or or so of the bench centers around the DX9 feature set.)

The purpose of the bench is not to simulate games but to compare and contrast vpus on a level, DX9-centered playing field, IMO. If one manufacturer's driver cheats the bench by doing less work while running the bench than another manufacturer's driver, then the bench is no longer valid as a comparison since there is a difference in the workload each vpu is doing while running it. The same principle would apply to a fly-by demo, a time demo, or any other comparative performance-ranking tool which features a camera on a track. Unless the workload undertaken by the compared vpus is identical, the fps results of running the software cannot be used in a performance comparison.

exactly, thats what i was getting at

 

[Xmas]

No benchmark can match real life performance, but they should make every effort to do so!

Something like rendering the scene in the wrong order, which is what that particular benchmark did, pretty much make it worthless.

In the wrong order? You mean sky first? Do you realize that nearly every game does this?

Well, it saves the Z clear
I'm sure a lot of these games would run faster rendering the sky last (without z write btw), at least on newer hardware.

 

[Doomtrooper]

It is synthetic scripted tests using no known game engine, you can't pick it up and run around in Game Test 4 in a first person shooter environment multiplaying with other people

So is UT 2003 Botmatch, scripted and you can't play in the Botmatch.

3dmark isn't sold as game, its sold as a benchmark

UT 2003 Botmatch Demo is not a game but a benchmark, which BTW is testing nothing for future engines, not even Pixel Shaders.

 

[jjayb]

You could apply what i'm saying to 3DMark03

but there is one main difference that I see, I was refering to a real game. 3DMark03 is not a game. It is synthetic scripted tests using no known game engine, you can't pick it up and run around in Game Test 4 in a first person shooter environment multiplaying with other people (though that would be kinda fun )

3dmark isn't sold as game, its sold as a benchmark

By the same token, showing q3 benchmarks or doom3 benchmarks doesn't tell me how well my card is going to run Dark age of camelot or IL-2 Sturmovik either. Seems that most sites use mainly first person shooters for benchmarks. What If I don't play first person shooters? How well is my card going to run Enclave?

3dmark03 performance does not equal Serious Sam performance. Ok, got that much, but what does Serious Sam benches show me? Serious Sam performance. Do Serious benches show me how well my card will perform in other games? Just because my card gets great bench scores in serious sam does that mean it's going to get great scores in other games? Maybe, mabye not. So how is that any more of an effective benchmark than 3dmark03?

Just because you can't "Play" 3dmark03 doesn't mean it can't be used as a tool to guage your cards performance in certain situations.

 

[OpenGL guy]

No benchmark can match real life performance, but they should make every effort to do so!

Something like rendering the scene in the wrong order, which is what that particular benchmark did, pretty much make it worthless.

In the wrong order? You mean sky first? Do you realize that nearly every game does this?

Well, it saves the Z clear

No, it doesn't. Sky boxes are often drawn without Z being enabled, in fact, this is preferred. Also, many Z optimizations don't work if you don't clear the Z buffer.

I'm sure a lot of these games would run faster rendering the sky last (without z write btw), at least on newer hardware.

This could be true.

 

[Xmas]

In the wrong order? You mean sky first? Do you realize that nearly every game does this?

Well, it saves the Z clear

No, it doesn't. Sky boxes are often drawn without Z being enabled, in fact, this is preferred. Also, many Z optimizations don't work if you don't clear the Z buffer.

That is true for today's hardware but I believe many older chips are faster without a separate Z clear.

I'm sure a lot of these games would run faster rendering the sky last (without z write btw), at least on newer hardware.

This could be true.

It's almost guaranteed to be true, as the only difference is that only visible sky pixels are rendered. How could anyone come up with such a ridiculous idea like rendering the sky first?
So, yes, I would also say it's the wrong order.

 

[Dio]

If you look at the list of applications that do render the sky first you might be surprised.

Why do you think it was a top bullet point in Richard Huddy's EGDC talks last year?