My response to the latest HardOCP editorial on benchmarks...

[Joe DeFuria]

http://www.hardocp.com/article.html?art=NDMwLDE=

At this moment in time, I find it hard to place any real-world value on the 3DMark03 score, as it does not represent anything but specific tests that FutureMark deems valuable.

And yet, you have not demostrated that you understand what the 3D Mark score represents. Furthermore, I believe that you have an issue not with what FutureMark deems valuable (who got input from all IHVs), but with what nvidia deems NOT valuable.

To put it simply, current synthetic benchmarks overall, do a disservice to the hardware community and everyone that will ever buy a 3D video card or a computer that has one installed.

Not necessarily true. To the extent that IHVs concentrate on "artifically" raising synthetic benchmarks, you are right. However, this applies to ANY benchmark, be it a "game" benchmark or a synthetic one.

I seem to recall a big stink made by HardOCP when some vendor had a GAME SPECIFIC path in their drivers that lead to gasp, optimization in a game benchmark....

ATI has had advance copies of the benchmark, where NVIDIA has not,

You really believe that? Or did the new "miracle 3DMark drivers" from nVidia magically appear the same day that 3DMark was released....without nVidia having prior access to some form of the benchmark?

this is because NVIDIA will no longer pay FutureMark a “subscription fee”.

Right. Not through any forcible measure from FutureMark.

The short answer to the question is that it does not benefit us at all, but rather harms us. While NVIDIA and ATI are slaving away to make sure that optimizations are built into their drivers so they get better benchmark scores as illustrated above, the gamer's true experience gets ignored.

Again, gross oversimplification. (Just because drivers run a synthetic benchmark better, does not mean they do that to the exclusion of "real games" getting better performance as well.)

I certainly agree that there is a very skewed over reliance on certain benchmark scores. Where I disagree, is who's PROBLEM this really is, and what the remedy is.

We have been reaching out to the game developers for a long time to ask them to include the right tools to allow us to natively benchmark popular games.

This is a good thing.

Ask a GPU/VPU maker and they will tell you that they have to optimize for synthetic benchmarks. That is how their products get rated in print and web publications.

Bingo.

The fault lies with PRINT AND WEB PUBLICATIONS that mis-use the benchmarks. INGNORING synthetic benchmarks is not the solution. Recongizing what the the synthetic benchmarks represent, and using them accordingly is the answer.

This is why we see “point and click” benchmarks that give you a one number result have so much impact. Sometimes simplicity seems to outweigh data value in benchmarking.

Again, agreed. But again, the solution is not to IGNORE the "one number results", but to understand what they represent, and use the results in proper context.

Overall the ultimate responsibility of giving benchmarks traction falls into the laps of every computer hardware reviewer and editor in the world. We are the ones that give benchmarks credence and value in the community.

Agreed.

If we use bad tools, so will everyone else. If editors and other decision makers are using these tools and basing conclusions on their results, companies like ATI and NVIDIA have no choice but to allocate huge resources to make sure they score well.

Again, the question is whether the tool is "bad", or if it's being used improperly. Using a tool improperly is just as bad or worse than using a bad tool in the first place.

Taking a tool, and just "not using it" because it can be misused is nonsense. ANY benchmark tool...even those based on actual, real, shipping games, can be misused.

Maybe separate benchmarks to pick from that are based on games that are shipping or are currently in development is what is needed for a proper evaluation process.

See Gameguage.

Now of course, you have developers "optimizing" for these specific games to squeeze out that extra 2 FPS, even if there are other games that need more attention.

You're also going to get a LOT of "pushback" from most developers concerning using "games in development" for testing. Games in "development" do not represent the final performance of the game (probably not as optimal with performance), and no developer wants their game performance "pre-judged" based on pre-release benchmarks.

Maybe we need an organization with a logo that can be included on game boxes so you know the game you are buying is part of the solution in getting better products to your hands.

Sure...how about "Nvidia...the way it's meant to be played!"

Basically, you have not given any solution to the problem of "how do we get an indication of how this hardware run TOMORROW'S games?" All the benchmarking of "today's games" don't mean squat if "tomorrow's games" are significantly different.

Having an "independent 3rd party" (like FutureMark), come up with a BEST GUESS, based on IHV input, seems like a reasonable course of action.

In short, just IGNORING synthetic benchmarks is not the solution, that does a disservice to the consumers you claim you are out to protect. Putting synthetic benchmarks in the proper context in a review is.

 

[KimB]

Re: My response to the latest HardOCP editorial on benchmark

I'm not Kyle, but I figure I'll put in my own two cents here.

And yet, you have not demostrated that you understand what the 3D Mark score represents. Furthermore, I believe that you have an issue not with what FutureMark deems valuable (who got input from all IHVs), but with what nvidia deems NOT valuable.

That first part is the entire point. Nobody understands what the 3DMark score actually represents.

Not necessarily true. To the extent that IHVs concentrate on "artifically" raising synthetic benchmarks, you are right. However, this applies to ANY benchmark, be it a "game" benchmark or a synthetic one.

I seem to recall a big stink made by HardOCP when some vendor had a GAME SPECIFIC path in their drivers that lead to gasp, optimization in a game benchmark....

Right. I've always said that I'm against any application-specific optimizations. I definitely agree with this. This basically means that nVidia (and ATI) is going to be sepending lots of time optimizing for this benchmark in such a way that no games benefit from the driver changes. This makes for a waste of time for nVidia and ATI, meaning some work is going to need to be diverted from real games. So while I think specific optimization for a game benchmark is bad enough (though I do accept that it is necessary), specific optimization for a non-game benchmark is really bad (for gamers).

You really believe that? Or did the new "miracle 3DMark drivers" from nVidia magically appear the same day that 3DMark was released....without nVidia having prior access to some form of the benchmark?

nVidia withdrew from the beta in December.

The fault lies with PRINT AND WEB PUBLICATIONS that mis-use the benchmarks. INGNORING synthetic benchmarks is not the solution. Recongizing what the the synthetic benchmarks represent, and using them accordingly is the answer.

I think it would be a good thing if somebody (nVidia or ATI...) actually got fed-up and attempted to kill off synethetic-benchmarks-that-try-to-be-game-benchmarks. It would be a risk, but it might be done if they actually deoptimize for that particular benchmark in the drivers, making it obvious to huge numbers of people that the non-game benchmark really is a terrible measure of real game performance, and forcing companies like Futuremark to try harder to use some real-game situations (since if nVidia or ATI took a stand on this, Futuremark would be screwed).

Again, agreed. But again, the solution is not to IGNORE the "one number results", but to understand what they represent, and use the results in proper context.

But nobody knows what they mean, not as it relates to how games will play ever.

See Gameguage.

Now of course, you have developers "optimizing" for these specific games to squeeze out that extra 2 FPS, even if there are other games that need more attention.

Which will always be a problem. There is no really good solution. The only possible solution is to benchmark every game, but that's just not feasible for any reviewer. The best we can hope for here is that the popular games will be optimized for in a non-application-specific manner, which will, hopefully, lead to optimizations in game X also helping with game Y.

Basically, you have not given any solution to the problem of "how do we get an indication of how this hardware run TOMORROW'S games?" All the benchmarking of "today's games" don't mean squat if "tomorrow's games" are significantly different.

Nobody can make that distinction, least of all a company like Futuremark. The best you can do is speculate that the advanced features of the later generations will improve performance noticeably. Saying anything more really is baseless.

 

[Typedef Enum]

If NVIDIA and ATI are devoting resources to optimizing their drivers for a synthetic benchmark that does not equate to real world gaming, what benefit is in it for us? By “usâ€￾, I mean the folks that buy the video cards and the games. The short answer to the question is that it does not benefit us at all, but rather harms us. While NVIDIA and ATI are slaving away to make sure that optimizations are built into their drivers so they get better benchmark scores as illustrated above, the gamer's true experience gets ignored

Seriously...Where have I heard this before? I know it sounds familiar...

Ah, yes...

"The reason that we're not all gung ho about it is that (3DMark'03) is not representative of (actual) games, nor is it a good benchmark," said Tony Tamasi, senior director of desktop product management at Nvidia. "That means Nvidia has to expend effort to make sure it runs well on our hardware. All that energy that we spend doesn't benefit the user. None. Zero. All that effort doesn't go to benefit any game, either. That's kind of depressing."

Yeah...That sounds like somebody who is speaking their mind...It didn't sound like the guy just got off the phone with nVidia, detailing all the reasons why this benchmark sucks.

 

[Joe DeFuria]

Nobody understands what the 3DMark score actually represents.

And I place a large part of the blame on FutureMark (in another thread) for that.

However, I figured out what the benchmark's purpose was simply by reading the white paper and using common sense. (Game scores = 0 if such and such feature is not supported.) I would think any other web site that purports to be "in tune with 3D hardware" would be able to reach a similar conclusion.

This basically means that nVidia (and ATI) is going to be sepending lots of time optimizing for this benchmark in such a way that no games benefit from the driver changes.

You can't claim that as an absolute.

nVidia withdrew from the beta in December.

I know. Do you think nVidia didn't have a late beta copy of 3DMark at that time?

I think it would be a good thing if somebody (nVidia or ATI...) actually got fed-up and attempted to kill off synethetic-benchmarks-that-try-to-be-game-benchmarks.

I think it would be a bad thing to kill off synthetic benchmarks that try-to-encompass-what-future-games-will-look-like-given-input-from-any-IHV-willing-to-participate.

That is, unless there is another way to try and guess what the "ability to play" future games is, without actually traveling through time into the future.

and forcing companies like Futuremark to try harder to use some real-game situations.

Again...please show us a "real game situation" as it exists in July, 2004.

The best we can hope for here is that the popular games will be optimized for in a non-application-specific manner, which will, hopefully, lead to optimizations in game X also helping with game Y.

We can also hope that forward looking benchmarks are optimized in a non spplication specific manner which will, hopefully, lead to better "driver readiness" for games in the future.

Nobody can make that distinction, least of all a company like Futuremark.

Most of all a company like Future mark, who is IHV indepdendent, and simply uses the latest DirectX features in a way that improves visuals.

The best you can do is speculate that the advanced features of the later generations will improve performance noticeably. Saying anything more really is baseless.

No, you can also speculate how the features will be used, which is what FutureMark does with the aid of IHVs.

Look, instead of arguing over "intentions" and "ability to predict" etc, why don't we actually look at the current RESULTS of 3DMark'03? And use that as a measure to guess how well FutureMark's results correlate to our own expectations?

Can we see if the results actually correlate to what we would consider the "Overall Goodness" of the videocards in question, with respect to how we anticipate they will be able to handle "future games?"

I would say that "for the future", the cards should be ranked in the following way, based on their "ability to handle future games".

1) DX7 cards like Radeon 7xxx, GeForce2/4 MX. All grouped together at the bottom of the heap.

2) DX8 cards like Radeon 8500, GeForce3/4. A step change above the DX7 cards. Radeon 8500 should probably fall somewhere above the GeForce3 series, and below or near the bottom section of the GeForce4 Ti series.

3) DX9 cards with "moderate" performance, like GeForceFX non-ultra, and Radeon 9500/9500 Pro. A step change above the DX8 cards.

4) "Top of the line" DX9 cards like the FX Ultra and Radeon 9700. A step above the lower DX9 cards...but in a predictable way based on pixel filling performance.

Agree or disagree with those predictions?

 

[Morris Ital]

I'm still worried about having to draw more polygons with a slower card, this doesn't seem logical. It should be at least the same, even if you have to make more passes.

??

:? :?

 

[Joe DeFuria]

See my other post. There are not "more polygons" in the scene. It's just that less flexible architectures must re-render the SAME polygons multiple times, in order to apply the same effects that a more advanced architecture can do "all at once."

 

[Nappe1]

Re: My response to the latest HardOCP editorial on benchmark

That first part is the entire point. Nobody understands what the 3DMark score actually represents.

COME ON!
Everyone knows that whoose got best 3DMarks has longest virtual <you definately know what> (!!)

so, problem solved.

 

[RussSchultz]

I'd be more comfortable with the 'futureness' of the benchmark if they had more different engines and techniques, and presented the numbers as "now, immediate future, and our guestimate of the far future".

As it stands now, it returns a number that is high if you match up with their singular vision of the future. Of course, presumably their vision takes input from IHVs and developers (or just spent some time perusing Carmack's .plans).

The number of 'goodness' points you have is really only certainly on target for the first 1/3rd, probably on target for the 2nd 1/3rd and just a shot i the dark for the last portion.

 

[OpenGL guy]

I'm still worried about having to draw more polygons with a slower card, this doesn't seem logical. It should be at least the same, even if you have to make more passes.

If you have to draw the same polygon five times to achieve an effect than that counts as five polygons.

 

[Mark]

If NVIDIA and ATI are devoting resources to optimizing their drivers for a synthetic benchmark that does not equate to real world gaming, what benefit is in it for us? By “usâ€￾, I mean the folks that buy the video cards and the games. The short answer to the question is that it does not benefit us at all, but rather harms us. While NVIDIA and ATI are slaving away to make sure that optimizations are built into their drivers so they get better benchmark scores as illustrated above, the gamer's true experience gets ignored

Seriously...Where have I heard this before? I know it sounds familiar...

Ah, yes...

"The reason that we're not all gung ho about it is that (3DMark'03) is not representative of (actual) games, nor is it a good benchmark," said Tony Tamasi, senior director of desktop product management at Nvidia. "That means Nvidia has to expend effort to make sure it runs well on our hardware. All that energy that we spend doesn't benefit the user. None. Zero. All that effort doesn't go to benefit any game, either. That's kind of depressing."

Yeah...That sounds like somebody who is speaking their mind...It didn't sound like the guy just got off the phone with nVidia, detailing all the reasons why this benchmark sucks.

I said the exact same thing to myself when I read that on [H]. I hate to say things like this on this forum, but it seems to me that Kyle is pretty gullible.

 

[Jima13]

If NVIDIA and ATI are devoting resources to optimizing their drivers for a synthetic benchmark that does not equate to real world gaming, what benefit is in it for us? By “usâ€￾, I mean the folks that buy the video cards and the games. The short answer to the question is that it does not benefit us at all, but rather harms us. While NVIDIA and ATI are slaving away to make sure that optimizations are built into their drivers so they get better benchmark scores as illustrated above, the gamer's true experience gets ignored

Seriously...Where have I heard this before? I know it sounds familiar...

Ah, yes...

"The reason that we're not all gung ho about it is that (3DMark'03) is not representative of (actual) games, nor is it a good benchmark," said Tony Tamasi, senior director of desktop product management at Nvidia. "That means Nvidia has to expend effort to make sure it runs well on our hardware. All that energy that we spend doesn't benefit the user. None. Zero. All that effort doesn't go to benefit any game, either. That's kind of depressing."

Yeah...That sounds like somebody who is speaking their mind...It didn't sound like the guy just got off the phone with nVidia, detailing all the reasons why this benchmark sucks.

I said the exact same thing to myself when I read that on [H]. I hate to say things like this on this forum, but it seems to me that Kyle is pretty gullible.

that or he knows what EFT stands for

 

[Morris Ital]

See my other post. There are not "more polygons" in the scene. It's just that less flexible architectures must re-render the SAME polygons multiple times, in order to apply the same effects that a more advanced architecture can do "all at once."

Both www.rage3d.com and [H]ardOCP report that there are more polygons as well as re-rendering multiple times. From the rage3d article:-

"With 1.1 pixel shaders, objects need one rendering pass for depth buffer initialization and three passes (stencil pass, light fall-off to alpha buffer, and diffuse and specular reflection) for each light source. When 1.4 pixel shader support is available only one pass for each light source is needed.

Approximately 250,000 polygons are rendered per frame using 1.1 pixel shaders
With 1.4 pixel shaders this number is lowered to approximately 150,000 "

From [H]


". This game test can use Pixel Shader 1.1 or 1.4 where supported in hardware. If a card capable of 1.4 pixel shader, it only needs to do one pass for depth buffer setup and a single pass per light source. But if it does not detect a PS 1.4 card then it uses PS 1.1 which uses one pass for the depth buffer but three passes per light source

Here is the kicker, if using Pixel Shader 1.4 it will render approximately 150,000 polygons, but if it has to utilize Pixel Shader 1.1 then it will render approximately 250,000 polygons."

Even if pixel shader 1.1 could do it in one pass it is being asked to do 66% more polygons .. why ?

If there is no easy answer why this is so then I think it is NOT wise to foster 3DM03 on the general public and have it for us geeks only.

The general public cannot even grasp XP performance ratings for Athlons, never mind pixel shaders / polygons for this.

Go on then , name me the actual number of Mhz for a Barton Xp2800 off the top of your head? You can't? Well look at this graph then, the Radeon 9700 Pro scores twice as much as a Gf4 4600 in 3DM03 so must be twice as good. Interest free credit .. now you're talking !

Wow, I even managed to go off topic in my own post ! LOL

 

[Ichneumon]

Even if pixel shader 1.1 could do it in one pass it is being asked to do 66% more polygons .. why ?

If there is no easy answer why this is so then I think it is NOT wise to foster 3DM03 on the general public and have it for us geeks only.

What you are not understanding is that 1.1 shader cards are rendering more polygons because they only support 1.1 shaders, not because they're being forced to do the extra work. They have to push more polygons through the pipe because 1.1 requires that they must do additional passes on each frame to achieve the effects in those tests.

Looking at it another way, 1.4 shader hardware is doing less work because they can collapse those extra passes that 1.1 shaders need to do into a single pass. They render less polygons because they have less passes per frame.

 

[Dave H]

Both www.rage3d.com and [H]ardOCP report that there are more polygons as well as re-rendering multiple times.

Well they're both wrong. Considering all the "3dMark03 reviews" have just been exact paraphrases of the white paper (even the one at B3D, although at least it paraphrased it more correctly), let's just quote from the source itself:

"Approximately 250,000 polygons are rendered per frame using 1.1 pixel shaders. With 1.4 pixel shaders this number is lowered to approximately 150,000 due to reduced number of rendering passes." (emphasis mine)

Considering most (but apparently not all in game2) polys undergo two passes with PS1.4 and four with PS1.1, these numbers (along with the 560,000/280,000 numbers for game3) are entirely expected.

Go on then , name me the actual number of Mhz for a Barton Xp2800 off the top of your head? You can't?

Barton 2800+ is clocked at 2 GHz. I think. The part was just officially released but was not benched anywhere and generally overlooked in review discussions in favor of the Barton 3000+. Thoroughbred 2800+ is 2.25 GHz.

(Note: just checked; the Barton 2800+ is said to be 2.083 GHz, which is complete nonsense because that means AMD has decided to award 200+ for adding 256k L2 to a 2600+, but 300+ for making the same change to a 2700+. In any case, this just proves the point: no one understands AMD's rating system because there's nothing to understand. It has nothing to do with performance, only marketing and price structure.)

Well look at this graph then, the Radeon 9700 Pro scores twice as much as a Gf4 4600 in 3DM03 so must be twice as good.

A Radeon 9700 Pro IS twice as good as a GF4 Ti4600. (Unfortunately, it scores 3 times as high in 3dMark03. )

 

[Morris Ital]

I'm still worried about having to draw more polygons with a slower card, this doesn't seem logical. It should be at least the same, even if you have to make more passes.

If you have to draw the same polygon five times to achieve an effect than that counts as five polygons.

But the futuremark claim is is 250 000 v 150 000 which is 1.6666667 times as much and you can't have 1.6666667 passes.

Another fact is that this benchmark is almost totally gpu/vpu dependant whereas most games as cpu limited to some extent ( see Anands scaling graphs for UT2003 for example). This is another good reason why [H] might want to drop it at present.

In the future it might be more representitive of what to expect but then Russ Schultz rightly states :-

"As it stands now, it returns a number that is high if you match up with their singular vision of the future"

Yep

 

[Morris Ital]

Even if pixel shader 1.1 could do it in one pass it is being asked to do 66% more polygons .. why ?

If there is no easy answer why this is so then I think it is NOT wise to foster 3DM03 on the general public and have it for us geeks only.

What you are not understanding is that 1.1 shader cards are rendering more polygons because they only support 1.1 shaders, not because they're being forced to do the extra work. They have to push more polygons through the pipe because 1.1 requires that they must do additional passes on each frame to achieve the effects in those tests.

Looking at it another way, 1.4 shader hardware is doing less work because they can collapse those extra passes that 1.1 shaders need to do into a single pass. They render less polygons because they have less passes per frame.

Nope. I'm still unconvinced that 150 000 1.4 polygons can go up to 250 000 1.1 polygons because of extra passes.

That's an extra 1.666667 passes which is not a whole number and so seems absurd.

I take it to mean that the 1.4 card does 150 000 polygons in one pass and the 1.1 card is forced to do 250 000 polygons FOR 4 passes or whatever.

Am I reading this wrong ?

Regards

Andy

 

[Ichneumon]

Edit: Answered better below.

 

[Doomtrooper]

Ignore Kyle, he is just the messenger ..he takes the Spoon Fed PR drivel from Nvidia..puts more twists on it and posts it as fact for more hits..like the Morton Downy Jr. Show of the internet.

I can only hope Brent brings some level headed unbiased sense to future reviews like he has done.
The comment that ATI is the only one with advanced copies is Horse Poop, Nvidia were part of the Beta team back in December and you could download the damn thing right off Nvidias FTP site

 

[Maverick]

Am I reading this wrong ?

Yes.

I'm just making these numbers up, but they do fit:

The scene has 50,000 polys.
Using 1.4 shaders, 3 passes are required to achieve the required result. 3 * 50,000 = 150,000.
Using 1.1 shaders, 5 passes are required. 5 * 50,000 = 250,000.

See?

 

[Doomtrooper]

If the hardware doesn't support PS 1.4 there isn't much you can do about it is there. Enough of the PS 1.4 whining, If futuremark inlcuded PS 1.3 then the Geforce 3 folks would be ranting and raving as Pixel Shader 1.1 isn't supported...can't please everyone.