[H] Benchmarking Future Ed.

Editor's Note: In an effort to simplify the key point behind all of this, let us examine the following scenario. Consider a situation where an editor is reviewing two graphics cards. Using a popular synthetic benchmark suite, the editor discovers that Card A has twice the Pixel Shader performance of Card B. As a result, the editor writes a blurb in the conclusion of the review, which outlines the apparent fact, that Card A will perform better than Card B in upcoming titles that rely upon that Pixel Shader. Roughly a month or so after the review is posted, Doom3 arrives on the market. The editor fires up the game on his testbed and begins benchmarking the cards once more. Imagine his surprise when he finds that Card B now has the sizeable performance advantage!

But the question is, does Doom3 even use pixel shaders at all? Not as far as I'm aware!

Doesn't exactly inspire confidence in the ability of [H] to clearly define what they are talking about.

3dfx made the frame rate comparison the de facto method for evaluating video cards. It seems ironic that some of 3dfx?s technology in the GeForceFX would hammer one of the first nails into the frame rate coffin.

What technology is this? :?
 
Mariner said:
But the question is, does Doom3 even use pixel shaders at all? Not as far as I'm aware!

Pretty sure it does, yeah. At least in the R200, NV30, and ARB2 codepaths. Not sure about the bottom-end codepath.

What technology is this? :?

Ummm, massive delays? :)
 
Doom3 is not a shader targetted title - its feture arget was DX7 style graphics chips (NV1x/R100). AFAIK there is only one shader in use and that is to collapse the number of passes required for the lighting model.
 
Editor's Note: In an effort to simplify the key point behind all of this, let us examine the following scenario. Consider a situation where an editor is reviewing two graphics cards. Using a popular synthetic benchmark suite, the editor discovers that Card A has twice the Pixel Shader performance of Card B. As a result, the editor writes a blurb in the conclusion of the review, which outlines the apparent fact, that Card A will perform better than Card B in upcoming titles that rely upon that Pixel Shader. Roughly a month or so after the review is posted, Doom3 arrives on the market. The editor fires up the game on his testbed and begins benchmarking the cards once more. Imagine his surprise when he finds that Card B now has the sizeable performance advantage!

The problem with all of this convoluted "logic" is that running a game which employs an optimized code path for certain hardware and comparing the results to other hardware running the same game without such an optimized code path in no way invalidates the benchmark which offers a comparison of both products, neither of which is supported through vendor-specific code paths.

Why? Because despite what Kyle believes, the vast majority of 3D games sold do not offer "optimized code paths" for anybody's hardware. Therefore, the benchmark which offers no vendor-specific code paths is actually more accurate generally in assessing likely performance than the single game which offers vendor-specific pathing for one product but not any others. Isn't this fairly obvious?

I think Kyle's entire 3D software world can be boiled down to 3D Mark and D3, only one of which is currently shipping...;) Hey, Kyle, it's a GIGANTIC 3D software world out there.... :!: (And besides, it's way too soon to hypothesize D3 performance comparisons. Why you think your nv35 D3 demo has any meaning apart from a nVidia-sponored nV35 PR campaign beats me.)



3dfx made the frame rate comparison the de facto method for evaluating video cards. It seems ironic that some of 3dfx?s technology in the GeForceFX would hammer one of the first nails into the frame rate coffin.

Hopefully, Kyle is not so twisted by all of this that he imagines that people are going to stop looking at frame-rate/IQ comparisons simply because nVidia no longer deems it convenient... I think Kyle is smokin' the same stuff they're passing around at nVidia these days....;)

The real danger to Kyle with all of the nonsense he's spouting here is that people will begin to see how little he apparently understands about the very topics on which he opines... :idea:

Kyle, I really think you need to substantially expand your 3D software environment. Man does not live on D3 previews/3D benchmarks alone, you know... The entire situation can be boiled down to this:

Q: Why, after all these years, is nVidia seeking to discredit frame-rate benchmarks?

A: Because nVidia's competitors currently run them faster with better IQ.

Simple as that.
 
That dood either need to stop doing drugs or start doing drugs because something is wrong with his brain. Let me get this straight he's claiming a DX9 synthetic benchmark Pixel Shader results do not tell you how well a Doom3 (OpenGL with little Pixel Shader usage at all) will perform so it is useless.
The drivel divergence of technology is mind boggling. Nvidia is purposefully causing the divergence. The whole article is poop.
 
Sad.

I'm just disapointed in myself for even reading the first page of that editorialmercial and giving that tool Kyle even 1 hit from me. :(

HOW can he keep basing the entire industry on a game that is not only not out yet but hasn't even had an announced shipping date yet?!?

I can't think of a whole lot of games out there that have optimized paths for different cards, I really can't. How can he be passing this off as accurate?

More importantly, how the hell could Pelly have his name in the byline and how can Brent live with himself working for such a schmuck? :(

[T]ardOCP sucks, but they win a capital 'T' out of me for this one at least. ;)
 
Editor's Note: In an effort to simplify the key point behind all of this, let us examine the following scenario. Consider a situation where an editor is reviewing two graphics cards. Using a popular synthetic benchmark suite, the editor discovers that Card A has twice the Pixel Shader performance of Card B. As a result, the editor writes a blurb in the conclusion of the review, which outlines the apparent fact, that Card A will perform better than Card B in upcoming titles that rely upon that Pixel Shader. Roughly a month or so after the review is posted, Doom3 arrives on the market. The editor fires up the game on his testbed and begins benchmarking the cards once more. Imagine his surprise when he finds that Card B now has the sizeable performance advantage!

I think we have always said that its essential that, when using synthetic benchmarks, the reviewer has to know what he is doing and what conclusions can and can not be made. If your sysnthetic benchmark shows that pixel shader performance of board A is much better than board B then no game out there will change that fact. The only thing that can happen is that a different feature gets stress (for example stencil fillrate rather than pixel shader capability) or that a very different code path is implemented and used.

Skimping through the article I get the distinct impression that [H] thinks its going to be fine to compare apples to pears since thats how you play games. While its correct that people play games completely different (all effects on with crap fps, all effects off with top fps, some where in the middle, etc its all in the players personal prefference) this is actually completely irrelevant when judging game hardware. You need to make a choice when judging hardware : apples with apples, or pears with pears, etc depending on what game settings and paths you use. When you want to judge games and game playaibility then you can change settings for different boards but... if there is a clear visual difference or clear code difference then you can no longer compare performance at this stage. All you can fairly say in this case is that card X and Y both deliver (or not) a playble and acceptable visual experience in game Z, you can not say that X is faster or better than Y since they will be different.

Just IMHO of course...

K-
 
Well, I will not visit the site so I can't read the whole article. But based on the blurb that was copied, here is my translation:

******

"Here at [H], we already have our pre-conceived notion of how cards A and B should run pixel shading games and applications, or future titles in general. We won't tell you where this preconceived idea came from, or provide you with any relevant data which supports our preconceived notion. You aren't smart enough to understand it.

In a nutshell, any tests that don't agree with our preconceived notion of performance, simply will not be shown by us here at [H]. That would just be counter productive and confusing, wouldn't it? We basically believe that you, our readership, are a bunch of mindless idiots. You cannot possibly be trusted with being given ALL the information, and making up your own mind and drawing your own conclusions.

Our mission at [H] is not to give you all the information possible to make an informed assesment. Our mission is to only give you whatever information there is that agrees with our unsupported yet preconceived idea of how things should turn out, and forget the rest.

Here at [H] we've done the thinking for you!"
 
Joe DeFuria said:
Well, I will not visit the site so I can't read the whole article. But based on the blurb that was copied, here is my translation:

******

"Here at [H], we already have our pre-conceived notion of how cards A and B should run pixel shading games and applications, or future titles in general. We won't tell you where this preconceived idea came from, or provide you with any relevant data which supports our preconceived notion. You aren't smart enough to understand it.

In a nutshell, any tests that don't agree with our preconceived notion of performance, simply will not be shown by us here at [H]. That would just be counter productive and confusing, wouldn't it? We basically believe that you, our readership, are a bunch of mindless idiots. You cannot possibly be trusted with being given ALL the information, and making up their own mind and drawing your own conclusions.

Our mission at [H] is not to give you all the information possible to make an informed assesment. Our mission is to only give you whatever information there is that agrees with our unsupported yet preconceived idea of how things should turn out, and forget the rest.

Here at [H] we've done the thinking for you!"
Actually that's pretty bang-on right from the page I read, but much more honest.
 
Joe DeFuria said:
Well, I will not visit the site so I can't read the whole article. But based on the blurb that was copied, here is my translation:

Heh-Heh...Pretty funny...;) I couldn't resist adding my enlargements in brackets:

******

"Here at [H], we already have our pre-conceived notion of how cards A and B should run pixel shading games and applications, or future titles in general. We won't tell you where this preconceived idea came from, or provide you with any relevant data which supports our preconceived notion. You aren't smart enough to understand it. [And we know this because if we at [H] aren't smart enough to understand it there's no way you are.]

In a nutshell, any tests that don't agree with our preconceived notion of performance, simply will not be shown by us here at [H]. That would just be counter productive and confusing, wouldn't it? We basically believe that you, our readership, are a bunch of mindless idiots. You cannot possibly be trusted with being given ALL the information, and making up their own mind and drawing your own conclusions. [We at [H] want to ensure that we remain as mindless and idiotic as our audience and so we work hard to stay that way--all for you.]

Our mission at [H] is not to give you all the information possible to make an informed assesment. Our mission is to only give you whatever information there is that agrees with our unsupported yet preconceived idea of how things should turn out, and forget the rest. [Well-considered information is at best a burden we at [H] want to lift from your shoulders, and we're proud of how we reinterpret and reorder, "optimize" if you will, information so that it is suitable for the digestion of the common man.]

Here at [H] we've done the thinking for you! [To that end we have decided to publish a Mac-centered, Apple-oriented web site in addition to our current pages, for we share many philosophical values with the Apple corporation.]"
 
That is how we think NVIDIA skated on their 3DMark 03 “optimizationsâ€￾ with a pack of lawyers clearing the way. NVIDIA is optimizing for benchmarks quite clearly and that is something we will have to deal with. It seems certain that they are not alone in doing that, but from what information we have been privy to in the last month it certainly seems to us that NVIDIA is doing a bit more optimizing than their competition.
The above quote is something I wasnt expecting him to say.
 
micron said:
That is how we think NVIDIA skated on their 3DMark 03 “optimizationsâ€￾ with a pack of lawyers clearing the way. NVIDIA is optimizing for benchmarks quite clearly and that is something we will have to deal with. It seems certain that they are not alone in doing that, but from what information we have been privy to in the last month it certainly seems to us that NVIDIA is doing a bit more optimizing than their competition.
The above quote is something I wasnt expecting him to say.

It sounds like exactly what I'd expect him to say...He doesn't call some of what nVidia did "cheating" because he doesn't understand it well enough to see the glaring differences between what they did and what ATi did except to say that nVidia did "more of it." That's an inaccurate summation as ATi did nothing like some of the things nVidia did relative to 3DMark 03. Notice he says "we will have to deal with it" while categorically refusing to deal with it. It's the kind of "half-truth" attitude that so aptly characterizes the bulk of nVidia's PR. He's clearly been engulfed by it.
 
WaltC said:
micron said:
That is how we think NVIDIA skated on their 3DMark 03 “optimizationsâ€￾ with a pack of lawyers clearing the way. NVIDIA is optimizing for benchmarks quite clearly and that is something we will have to deal with. It seems certain that they are not alone in doing that, but from what information we have been privy to in the last month it certainly seems to us that NVIDIA is doing a bit more optimizing than their competition.
The above quote is something I wasnt expecting him to say.

It sounds like exactly what I'd expect him to say...He doesn't call some of what nVidia did "cheating" because he doesn't understand it well enough to see the glaring differences between what they did and what ATi did except to say that nVidia did "more of it." That's an inaccurate summation as ATi did nothing like some of the things nVidia did relative to 3DMark 03. Notice he says "we will have to deal with it" while categorically refusing to deal with it. It's the kind of "half-truth" attitude that so aptly characterizes the bulk of nVidia's PR. He's clearly been engulfed by it.
if he would have actually used the word "cheat", would it have made a difference in your eyes? I think it's a step in the right direction for him.
 
Back
Top