HardOCP's position on the 3DMark2003/Nvidia issue

That not a rigorous analysis.

Company B is a scoundrel because it cheated on the benchmark out of its selfish interest.

The benchmark is worthless because it does not represent realistic applications.

Since the benchmark is worthless, it may actually turn out that company B's cheats will lead to consumer to a better purchasing decision.

Convoluted? Yes! Wrong? No!

Does the consumer really care that company B is a scoundrel? Not really, the consumer just wants to buy a good video card, just like the way the consumer continues to buy MS software, use Worldcom for phone service, pump gass from Chevron, buy cookies from Philip Morris, etc.

 

I won't argue that with you at the moment, but I will ask you to consider why it is that nVidia's objection to the benchmark is that it doesn't accurately represent the way 3D games will be written in the future. At least, that's what nVidia gave as the reason it pulled out of the program. If it is not a future-based bench, just for the sake of argument, then you have described it as a backwards-looking program. So what is nVidia's objection to it based on?

Actually, "simulating games" is something 3DMark does, apart from a "goal." FutureMark has made it plain that '03 is a vpu benchmark and not the type of system benchmark that '02 certainly was. The goal of '03 is to primarily guage vpu performance, at this stage no one but nVidia sems to have a problem with it. If that problem nVidia has is not '03's approach to DX9, then what on earth could it be?

 

if you havent already got one I suggest you seek a job in nVidia's PR department

 

I really don't get this stuff about 3dmark. 3dmark is not a game . It is a tool to determine 3d performance relative to other cards. Please enlighten me and show me where 3dmark consistantly gets this wrong?
Has any ever got a 3dmark score that showed a GF4MX beating a GF4-4600? I suspect the anwser is no. Saying 3dmark is invalid becuase i doesn't represent an actual game is bunk. If it did represent an actual game ppl. would say that game is coded this way or that and is not representitive of some other game.
Lets' try an analogy.
I build a custom 2 mile track with all sorts of ups and downs and left handed turns etc.. then I drive Jeff Gordons car on it and it beats my Nissan. After that someone says well Nascar tracks are all ovals so the test is inaccurate cause tracks aren't built like that. Does that mean my Nissan is faster than Jeff Gordan's car>?
Of course not. All I see is people saying the test is worthless and have yet to see any evidence where it gets it wrong. If you wanna see how your card does in Q3 , test in Q3. If you want to know how your card will perform generally across all games use 3dmark.

 

LOL, what would they want with a physicist/material scientist?

I'd prefer being a lawyer over a PR rep in any case.

I don't see why anyone would passionately attack one company over another. Malfeascence abound in the corporate world, it's human nature.

All I really care about is getting a good video card to play DOOM III. That and having fun poking people who put moral credence into video game benchmarketing.

 

First of all, you're wrong in the premise of UT2K3 demos.

There are two types of UT2K3 demos: flybys and botmatches. They are both designed to do different things. "Flybys" do NOT represent actual gameplay performance. In fact, it's quite the opposite. The FlyBy demos are meant to NOT represent gameplay performance, by forcing GPU limitations to a far greater extent than actual gameplay. (Which bot-matches are suppoed to more or less represent.)

The "purpose" of the UT2K3 "Flyby" type demos are more or less EXACTLY like the purpose of 3DMark. To stress the GPU. FULL STOP. The main difference is:

1) UT is really a DX7 engine, not a DX8/DX9 engine, so UT will stress the GPUs differently than 3DMark.
2) The UT test uses an actual game engine, and 3D Mark doesn't.

But again, I stress that UT Flyby demos are very much the same as 3DMArk 03 in the sense that they do not represent "actual gameplay performance." They try and represent GPU performance.

You can no more predict "actual gameplay performance" of UT or any other DX7 game with a UT2K3 flyby demo, than you can predict actual gameplay DX8/9 gameplay performance with 3DMark.

So what good are they?

As we know, sometimes actual gameplay performace is not particularly GPU limited. Running a bunch of benchmarks that have heavy reliance on CPU or other non GPU related things, doesn't tell us much about the performance differences between GPUs.

That's exactly why there have always been "BotMatch" and "FlyBy" demos for UT. Both are useful...they just tell us different things.

That definition is incomplete. This definition of "optimizing" would indicate that the 3DMark clipping is an optimization, not a cheat. It is a cheat (based on the Extremetech analysis), and here is why....and most people aren't looking at it in the correct way:

If the basis of your optimization requires to to have access to data that is NOT PASSED by the game engine in real time, then that optimization is a cheat. This 3DMark cheat is based on the fact that the drivers "are told" the camera path won't change from some determined path. Problem is, they are not told this by the game engine. Clipping planes are inserted based on this knowledge. That data (the clipping planes) are not passed from the engine in real-time, nor are those planes calculated in real-time (as evidenced by the lack of correct rendering when "off the rail".)

That is why this is particular exanple is a cheat, and not a legal optimization. It relies on data that is not given by the benchmark, or calculated in real-time from data given by the benchmark.

This is why something like a "deferred renderer" is NOT cheating. It's not drawing "everything" either. But it calculates, on the fly, frame by frame, what is needed to be drawn. If you took a deferred renderer "off the rail" it would not suffer the clipping issues.

 

boobs, your hypothetical situation about 'company b and benchmark x' is just hypothetical. There's no real world situation like it going on...

 

Well said. With respect to other programs used to test various aspects of video cards (shader performance, AA, etc.) They are not games. Are you, boobs, suggesting that they provide no relevant info. inregards to performance ?

 

Right. Which was to misrepresent the performance of its products in order to influence sales.

This is where you aren't coming in clearly. The benchmark is a benchmark in this case and is not a real-world application. This is true of all benchmarks. 3D Mark is not unusual or an exception in that regard.

If your suggestion is to throw out all benchmarks, all time demos, and all commercial demos (such as nVidia's recent Doom III demo), I can't say I would disagree with that. However, I think you'll agree with me that this is not likely to happen... There is nothing in 3D Mark 03 to set it apart from any of these other kinds of benchmarks. Why is it OK for nVidia to pay for its own Doom III demos and foist those on an unsuspecting public, but it's not OK to run 3D Mark 03 because it is "worthless"...? That certainly seems to be [H]'s position in a nutshell.

Therefore, if 3D Mark 03 is "worthless" than so are all of them "worthless," including nVidia's recent Doom III demo. Or do you think the only benchmarks that should be thrown out are those nVidia deems worthless...?....

How so--if the consumer is mislead into buying an underperforming product?

The point you are missing is that what nVidia has done here is misrepresent a product--throwing out 3D Mark or not has absolutely nothing to do with it, since the people who like 3D Mark aren't going to quit using it because nVidia's drivers cheat it. Rather, they will be tempted to throw out nVidia, instead...

I think you can be assured you are pretty much speaking for yourself here, as most "consumers" take a dim view of dealing with scoundrel corporations who brazenly insult their intelligence and attempt to rip them off at the same time....

 

we need the Mother of All Benchmarks(TM) marks! it will run every game since 1995 at every resolution 3 times on a random demo path. Then it will be representive of real games (screw you pre-95 !) I just hope I can get it installed before dx10 comes out!

 

Yeah, but they still buy the products.

No benchmark can match real life performance, but they should make every effort to do so!

Something like rendering the scene in the wrong order, which is what that particular benchmark did, pretty much make it worthless.

Worthlessness is in degrees, maybe I should follow the fad and put a worthlessness meter in my sig.

 

In the wrong order? You mean sky first? Do you realize that nearly every game does this?

 

You could apply what i'm saying to 3DMark03

but there is one main difference that I see, I was refering to a real game. 3DMark03 is not a game. It is synthetic scripted tests using no known game engine, you can't pick it up and run around in Game Test 4 in a first person shooter environment multiplaying with other people (though that would be kinda fun )

3dmark isn't sold as game, its sold as a benchmark

but i agree cheating in benchmarks shouldn't happen either, cheating is cheating no matter what the application

 

exactly, thats what i was getting at

 

Well, it saves the Z clear
I'm sure a lot of these games would run faster rendering the sky last (without z write btw), at least on newer hardware.

 

So is UT 2003 Botmatch, scripted and you can't play in the Botmatch.

UT 2003 Botmatch Demo is not a game but a benchmark, which BTW is testing nothing for future engines, not even Pixel Shaders.

 

By the same token, showing q3 benchmarks or doom3 benchmarks doesn't tell me how well my card is going to run Dark age of camelot or IL-2 Sturmovik either. Seems that most sites use mainly first person shooters for benchmarks. What If I don't play first person shooters? How well is my card going to run Enclave?

3dmark03 performance does not equal Serious Sam performance. Ok, got that much, but what does Serious Sam benches show me? Serious Sam performance. Do Serious benches show me how well my card will perform in other games? Just because my card gets great bench scores in serious sam does that mean it's going to get great scores in other games? Maybe, mabye not. So how is that any more of an effective benchmark than 3dmark03?

Just because you can't "Play" 3dmark03 doesn't mean it can't be used as a tool to guage your cards performance in certain situations.

 

No, it doesn't. Sky boxes are often drawn without Z being enabled, in fact, this is preferred. Also, many Z optimizations don't work if you don't clear the Z buffer.

This could be true.

 

That is true for today's hardware but I believe many older chips are faster without a separate Z clear.

It's almost guaranteed to be true, as the only difference is that only visible sky pixels are rendered. How could anyone come up with such a ridiculous idea like rendering the sky first?
So, yes, I would also say it's the wrong order.

 

If you look at the list of applications that do render the sky first you might be surprised.

Why do you think it was a top bullet point in Richard Huddy's EGDC talks last year?