Joe DeFuria
Legend
I think this deserves it's own thread, because I see lots of people making arguments against 3DMark03 based on the wrong premise. The wrong premise is that the 3DMark03 "score" is a performance measurement. Thus, if the relative 3D Mark scores don't show what we expect relative performance to be in "todays games", they don't make sense.
Wrong.
The 3DMark score is more abstract than that. Ever since 3DMark 2001, when it was decided to not score certain tests at all depending on the hardware capability, the 3DMark score attempts to measure, for lack of a better term, the "Goodness" of a 3D system.
There are two aspects of this version of 3DMark that seem to give people trouble. One is how to interpret the score, and the other is how severely GPU dependent the score is, relative to the CPU. I'll address each in turn.
3DMark Score
Now, to be clear, the issue surrounding this is not really the fault of those taking the wrong position. It's FutureMark's fault for presenting 3DMark as a "performance benchmark." From their WhitePaper:
http://520041101062-0001.bei.t-online.de//APC/3DMark03-Whitepaper.pdf
It seems to me that FutureMark doesn't even know how it's score should be interpreted.
They present it as merley a way to "measure 3D graphics performance".
That being said, I find FutureMark's approach to getting a "score" as completely valid, though a bit ambiguous. Everyone (marketers, consumers, etc.) would ideally like to have a "single number" that tells them which card / system is best. (Not that it's really possible to do such a thing....) So it's understandable for the primary goal of 3DMark to produce such a number.
We can all agree that performance is but one aspect of the "goodness" of a video card / system. Feature support (level of DX support particularly), is just as important when making an evaluation of how "good" a video card is. It's easy to incorporate "performance" into an overall score by measuring FPS, but how do you factor in the feature support?!
One way (the way which FutureMark chose) is to penalize cards that don't support certain features. This is a valid approach. Where things get touchy, is deciding which features count and which ones don't. And "how much" to penalize for the lack of such features.
Again from the whitepaper to arrive at the 3D Mark score, FutureMark normalizes the FPS scores of the 4 game tests on a high end system, and uses that as a basis for giving "weight" to each of the game tests.
Test 1, 2, and 3 each get 26.7%, and test 4 gets 20% of the final 3D Mark.
Thus, all else being equal
1) A card that only supports DX7 is only 33% as good as a card that supports both DX7 and DX8.
2) A card that only supports DX7 is only 26% as good as a card that supports DX7, DX8, and DX9.
3) A card that only supports DX8 80% as good as a card that supports DX8 and DX9.
Now, one can argue back and forth about how FutureMark came to those percentages, or if they should be different, etc. But certainly nothing seems unreasonble to me, considering this is a forward looking test. (Also as described in the white-paper.)
The bottom line is, the 3DMark score is NOT simply a performance number. It is an attempt to quantify BOTH PERFORMANCE AND FEATURES. I see nothing wrong with this approach...other than FutureMark failing to sufficiently advertise the benchmark in that way.
OK, on to HEAVY GPU LIMITATIONS.
Again, FutureMark made the decision, based on how they saw games evolving, that going forward, games using these forward looking features are going to be much more GPU than CPU limiting than today and in the recent past. This is entirely reasonable to me. The advent of pixel shaders is the equivalent, IMO, to the advent of hardware 3D rendering in the first place. At first, we couldn't get enough fill-rate...being stuck at 30 FPS at 640x480 on a Voodoo1. Then, fill-rate "advanced" to at this point, many / most games that are NOT using Shading techniques are highly CPU limited, or balanced between CPU/GPU.
But shading performance is so relatively slow, at this time, that I see a step change shift back to GPU limiting when these features are enabled. This is not just common sense, but IIRC, by what Carmack has been saying is true of Doom3, one of the first games to make heavy use of such features. (That the GPU is going to be the biggest factor for getting good performance, very unlike Quake3 where the CPU played a large role.) I'm trying to find his exact words, but I think it was in an interview, not his .plan file....
In any case, I hope this long-winded post gets some people to look at 3DMark in a different way....including some people at FutureMark.
Wrong.
The 3DMark score is more abstract than that. Ever since 3DMark 2001, when it was decided to not score certain tests at all depending on the hardware capability, the 3DMark score attempts to measure, for lack of a better term, the "Goodness" of a 3D system.
There are two aspects of this version of 3DMark that seem to give people trouble. One is how to interpret the score, and the other is how severely GPU dependent the score is, relative to the CPU. I'll address each in turn.
3DMark Score
Now, to be clear, the issue surrounding this is not really the fault of those taking the wrong position. It's FutureMark's fault for presenting 3DMark as a "performance benchmark." From their WhitePaper:
http://520041101062-0001.bei.t-online.de//APC/3DMark03-Whitepaper.pdf
It seems to me that FutureMark doesn't even know how it's score should be interpreted.
They present it as merley a way to "measure 3D graphics performance".That being said, I find FutureMark's approach to getting a "score" as completely valid, though a bit ambiguous. Everyone (marketers, consumers, etc.) would ideally like to have a "single number" that tells them which card / system is best. (Not that it's really possible to do such a thing....) So it's understandable for the primary goal of 3DMark to produce such a number.
We can all agree that performance is but one aspect of the "goodness" of a video card / system. Feature support (level of DX support particularly), is just as important when making an evaluation of how "good" a video card is. It's easy to incorporate "performance" into an overall score by measuring FPS, but how do you factor in the feature support?!
One way (the way which FutureMark chose) is to penalize cards that don't support certain features. This is a valid approach. Where things get touchy, is deciding which features count and which ones don't. And "how much" to penalize for the lack of such features.
Again from the whitepaper to arrive at the 3D Mark score, FutureMark normalizes the FPS scores of the 4 game tests on a high end system, and uses that as a basis for giving "weight" to each of the game tests.
Test 1, 2, and 3 each get 26.7%, and test 4 gets 20% of the final 3D Mark.
Thus, all else being equal
1) A card that only supports DX7 is only 33% as good as a card that supports both DX7 and DX8.
2) A card that only supports DX7 is only 26% as good as a card that supports DX7, DX8, and DX9.
3) A card that only supports DX8 80% as good as a card that supports DX8 and DX9.
Now, one can argue back and forth about how FutureMark came to those percentages, or if they should be different, etc. But certainly nothing seems unreasonble to me, considering this is a forward looking test. (Also as described in the white-paper.)
The bottom line is, the 3DMark score is NOT simply a performance number. It is an attempt to quantify BOTH PERFORMANCE AND FEATURES. I see nothing wrong with this approach...other than FutureMark failing to sufficiently advertise the benchmark in that way.
OK, on to HEAVY GPU LIMITATIONS.
Again, FutureMark made the decision, based on how they saw games evolving, that going forward, games using these forward looking features are going to be much more GPU than CPU limiting than today and in the recent past. This is entirely reasonable to me. The advent of pixel shaders is the equivalent, IMO, to the advent of hardware 3D rendering in the first place. At first, we couldn't get enough fill-rate...being stuck at 30 FPS at 640x480 on a Voodoo1. Then, fill-rate "advanced" to at this point, many / most games that are NOT using Shading techniques are highly CPU limited, or balanced between CPU/GPU.
But shading performance is so relatively slow, at this time, that I see a step change shift back to GPU limiting when these features are enabled. This is not just common sense, but IIRC, by what Carmack has been saying is true of Doom3, one of the first games to make heavy use of such features. (That the GPU is going to be the biggest factor for getting good performance, very unlike Quake3 where the CPU played a large role.) I'm trying to find his exact words, but I think it was in an interview, not his .plan file....
In any case, I hope this long-winded post gets some people to look at 3DMark in a different way....including some people at FutureMark.
