What does the 3DMark Score Represent?

I think this deserves it's own thread, because I see lots of people making arguments against 3DMark03 based on the wrong premise. The wrong premise is that the 3DMark03 "score" is a performance measurement. Thus, if the relative 3D Mark scores don't show what we expect relative performance to be in "todays games", they don't make sense.

Wrong.

The 3DMark score is more abstract than that. Ever since 3DMark 2001, when it was decided to not score certain tests at all depending on the hardware capability, the 3DMark score attempts to measure, for lack of a better term, the "Goodness" of a 3D system.

There are two aspects of this version of 3DMark that seem to give people trouble. One is how to interpret the score, and the other is how severely GPU dependent the score is, relative to the CPU. I'll address each in turn.

3DMark Score

Now, to be clear, the issue surrounding this is not really the fault of those taking the wrong position. It's FutureMark's fault for presenting 3DMark as a "performance benchmark." From their WhitePaper:

http://520041101062-0001.bei.t-online.de//APC/3DMark03-Whitepaper.pdf

3D graphics benchmarking allows users to accurately evaluate the performance of the newest 3D graphics software technology on the latest 3D hardware...Each PC component – motherboard, CPU, system memory, graphics card, etc. – has multiple possible manufacturers, making 3D graphics benchmarking very complex. The 3DMark series makes measuring 3D graphics performance simple.

It seems to me that FutureMark doesn't even know how it's score should be interpreted. :rolleyes: They present it as merley a way to "measure 3D graphics performance".

That being said, I find FutureMark's approach to getting a "score" as completely valid, though a bit ambiguous. Everyone (marketers, consumers, etc.) would ideally like to have a "single number" that tells them which card / system is best. (Not that it's really possible to do such a thing....) So it's understandable for the primary goal of 3DMark to produce such a number.

We can all agree that performance is but one aspect of the "goodness" of a video card / system. Feature support (level of DX support particularly), is just as important when making an evaluation of how "good" a video card is. It's easy to incorporate "performance" into an overall score by measuring FPS, but how do you factor in the feature support?!

One way (the way which FutureMark chose) is to penalize cards that don't support certain features. This is a valid approach. Where things get touchy, is deciding which features count and which ones don't. And "how much" to penalize for the lack of such features.

Again from the whitepaper to arrive at the 3D Mark score, FutureMark normalizes the FPS scores of the 4 game tests on a high end system, and uses that as a basis for giving "weight" to each of the game tests.

Test 1, 2, and 3 each get 26.7%, and test 4 gets 20% of the final 3D Mark.

Thus, all else being equal

1) A card that only supports DX7 is only 33% as good as a card that supports both DX7 and DX8.
2) A card that only supports DX7 is only 26% as good as a card that supports DX7, DX8, and DX9.
3) A card that only supports DX8 80% as good as a card that supports DX8 and DX9.

Now, one can argue back and forth about how FutureMark came to those percentages, or if they should be different, etc. But certainly nothing seems unreasonble to me, considering this is a forward looking test. (Also as described in the white-paper.)

The bottom line is, the 3DMark score is NOT simply a performance number. It is an attempt to quantify BOTH PERFORMANCE AND FEATURES. I see nothing wrong with this approach...other than FutureMark failing to sufficiently advertise the benchmark in that way.

OK, on to HEAVY GPU LIMITATIONS.

Again, FutureMark made the decision, based on how they saw games evolving, that going forward, games using these forward looking features are going to be much more GPU than CPU limiting than today and in the recent past. This is entirely reasonable to me. The advent of pixel shaders is the equivalent, IMO, to the advent of hardware 3D rendering in the first place. At first, we couldn't get enough fill-rate...being stuck at 30 FPS at 640x480 on a Voodoo1. Then, fill-rate "advanced" to at this point, many / most games that are NOT using Shading techniques are highly CPU limited, or balanced between CPU/GPU.

But shading performance is so relatively slow, at this time, that I see a step change shift back to GPU limiting when these features are enabled. This is not just common sense, but IIRC, by what Carmack has been saying is true of Doom3, one of the first games to make heavy use of such features. (That the GPU is going to be the biggest factor for getting good performance, very unlike Quake3 where the CPU played a large role.) I'm trying to find his exact words, but I think it was in an interview, not his .plan file....

In any case, I hope this long-winded post gets some people to look at 3DMark in a different way....including some people at FutureMark. ;)
 
What the score represents? A total loss with reality on behalf of Futuremark I'd say.

When they said this iteration would be more GPU limited they weren't kidding! :) We got one DX7 test using fixed-function T&L with lots of transparencies and some double-texturing (specular highlights on the aircraft's fusilage). Geometry density seems rather low, unless planes in the far distance uses the exact same model as when seen up-close.

Next two tests features high poly densities, pixel shading on all surfaces and TONS of stencils.

I find this very peculiar. I expected pixel shading to be used, but I didn't expect a full Doom3-style per-pixel lighting and shadows engine to be used. Especially not in TWO tests!

We can't ascertain the shading performance of a card when it is being bogged down with this many stencils and separate rendering passes. For f*cks sakes, my heavily overclocked GF3 runs tests 2 and 3 at 1-5 fps, that's just totally rediculous. What on EARTH are those mad onions doing in their code really? That's as slow as my AMD K6-III with a 64-bit Rage128 Pro vidcard ran the high-detail 3DMark 2001 tests, and that system was MUCH further away from the then high-end than my current P4/1.7+GF3 is now!



*G*
 
We got one DX7 test using fixed-function T&L

Actually, there's no fixed function T&L. It's vertex shaders 1.1.

with lots of transparencies and some double-texturing (specular highlights on the aircraft's fusilage).

The airplanes are quad textured....

I expected pixel shading to be used, but I didn't expect a full Doom3-style per-pixel lighting and shadows engine to be used. Especially not in TWO tests!

That, I mostly gree with. I think tests 2 and 3 are a bit too similar in functionality....almost might as well be the same test. I would've preferred one test with all the stenciling / shadows...and the other one of the DX8 tests similar to the Nature or Advanced shader test of 3D Mark 2001....
 
These quotes are from the different 3DMark readme's.

3DMark2000 said:
3DMark2000 is a benchmark that focuses on testing 3D performance of modern 3D accelerators in a "real world" environment. As most users will use their 3D accelerator for playing games, 3DMark focuses on these types of applications in most of its results.

The 3DMark product line is the most comprehensive benchmark platform available for testing future game performance. The input from our BETA program, game developers and users allows us to predict what is going to be important for future game playing. We are delighted to introduce 3DMark2000 as the next stage of benchmarking. 3DMark2000 pushes the performance barrier yet again and moves benchmarks into the next Millennium.

3DMark2000 is also a powerful tool for improving your 3D performance and it helps you make informed decisions when choosing new hardware.

3DMark2001SE said:
3DMark2001 SE is a diagnostics tool for measuring the 3D game performance of PCs. It is entertaining and easy to use, which makes it "must have" software for all home PC users interested in 3D games. Even a beginner PC user can get a game performance measurement with 3DMark2001 SE. For the more advanced users, 3DMark2001 SE offers a wide range of display settings and testing options for the benchmark run. The most skillful performance tweakers can get the latest news on PC performance upgrading through MadOnion.com's on-line services. There you can also challenge the fastest PCs in the world and get good performance tuning advice in the discussion forum or the on-line chat.

At the same time, 3DMark2001 SE is a reliable tool for professional hardware performance measurement and comparison. The Pro version of 3DMark2001 SE includes additional tools for pro-users, making their work easier, less time consuming and more fun. The Pro version of 3DMark2001 SE can be purchased on-line, allowing you access to all of the "Pro features".

3DMark2001 SE uses a real game engine (MAX-FX Technologyâ„¢, by Remedy Entertainment) in all tests. This means that the obtained performance level in the game tests is accurate. If additional time would be spent on implementing more content, interactivity and game storylines, all game tests could be made into real games. We have proven this by making a game demo of the first game test. The scene is the same as in the test, but this time you are behind the steering wheel.

3DMark03 said:
3DMark03 is the latest worldwide standard in advanced 3D gaming performance benchmarking from Futuremark Corporation. This unique program provides an accurate and consistent assessment of the ability of your PC to cope with the latest and future generations of 3D games. The detailed diagnostic tools allow the user to make reliable hardware judgments and help to inform decisions over upgrades.

The incredible scenes in the various tests are part of the routines that 3DMark performs to measure the 3D gaming performance, as well as provide you with details about more specific abilities of your PC. By using 3DMark03 and the Online ResultsBrowser, you can get:

A 3DMark score - a globally recognised and comparable measurement of the 3D gaming ability of your PC.
Detail results of the game and feature tests.
A comparison of your machine against thousands of others, from every corner of the globe!
An educated estimate as to how much better your system will perform with certain upgrades.

I just wanted to save everybody the browsing through the different readme's. :)

After reading them it would seem the focus of 3DMark is measuring the ability of your system to play current and future 3Dgames.
 
After reading them it would seem the focus of 3DMark is measuring the ability of your system to play current and future 3Dgames.

I would say that is indeed a pretty accurate assesment of that 3DMark actually is (measure the "ability" to play current and future games), but I think that message is lost in their own descriptions.

Mostly, because they continually characterize it first and foremost as a "performance benchmark." Look at the first sentence in each desrcription. "Performance" is always there, but not features....
 
Joe DeFuria said:
Actually, there's no fixed function T&L. It's vertex shaders 1.1.

Oh, that would explain why on the same system, a Radeon 7500 PCI is several times faster than a GeForce2 Ti.
 
Oh, that would explain why on the same system, a Radeon 7500 PCI is several times faster than a GeForce2 Ti.

Why would that explain anything? I mean the radeon 7500 does not support VS1.1 in hardware and still has to do it in software via the CPU. What advantage in 3dmark does the Radeon have over the GF2 GTS?
 
Joe DeFuria said:
Mostly, because they continually characterize it first and foremost as a "performance benchmark." Look at the first sentence in each desrcription. "Performance" is always there, but not features....
You are right. As features play a pretty big role (TnL in 2000, PS and VS in 2001, PS 2.0 and VS 2.0 in 03) in the weighing of their "3DMarks: globally recognised and comparable measurement of the 3D gaming ability of your PC" one could expect them mentioned a bit more often.
 
"with the latest and future generations of 3D games"

yeah sure... Unreal II surely runs @ 4fps :p
And so does Doom III leaked demo... (both run way faster, like 5 to 10 times)


It's not a nice tech demo, it's not a good bench, it's not a fun screensaver... what is it then ?
 
Why is vertex shaders used in game test 1? That's just silly, there's ZERO deformation of models going on in that test. Using vertex shaders just penalizes older DX7 hardware when their on-board T&L engines could have done exactly as good a job as vertex shaders.

And quad texturing on the planes, where?! I barely notice the specular highlights! (These are mainly visible on the bomber at the end when it rolls away with its engines on fire, I never even saw it on any of the other planes).

Quad textures seems like a total waste, there's no damage decals as far as I can see either. They might just as well be single textured really, it matters little during most of the test sequence.

I'm also curious why the fillrate test STILL uses transparent textures since that burns tons more framebuffer bandwidth. To get around all types of pixel rejection techniques, maybe?


*G*
 
Maybe ET is exaggerating the fact that the test does, in fact, rely heavily on T&L performance (7500 T&L hardware>>Geforce 2 T&L), regardless of VS 1.1 functionality in the code.
 
Radeon 7200 and 7500 are equipped with a third TMU and superior HSR over a GTS (HZ II on the 7500)thats about it that I can think of.
The 7500 also has the 8500's memory controller.
 
Ingenu said:
"with the latest and future generations of 3D games"

yeah sure... Unreal II surely runs @ 4fps :p
And so does Doom III leaked demo... (both run way faster, like 5 to 10 times)

Well, it's supposed to be a forward looking benchmark, to be used probably until DX9 is replaced by DX10. That's a fairly long time, viewed from the perspective of gfx-cards. And just because you don't have a card that runs the benchmark well, doesn't mean other people don't. Cards that get scores in the 4-5000 range obviously run the benchmark at a decent clip even today. And by, say summer 2004, most cards probably will. 3DMark helps drive that development. To what extent it reflects game performance better by that time remains to be seen.

Entropy
 
Joe DeFuria said:
yeah sure... Unreal II surely runs @ 4fps :p

Exactly. And Unreal II surely doesn't use DX8 pixel shaders for any significant and global effects either. That's the point.

I'm not sure to understand you on this one.

I was being sarcastic about the 'latest' and 'future' games, and I pointed to titles which are/will be popular, and neither run as slow as the benchmark test...

So what does it test really ? 3 years in the future games ?

What's the point, I change my hardware every 2/3 years, so I don't care how well future games will perform on my current system ...
 
Grall said:
Why is vertex shaders used in game test 1? That's just silly, there's ZERO deformation of models going on in that test. Using vertex shaders just penalizes older DX7 hardware when their on-board T&L engines could have done exactly as good a job as vertex shaders.
Well the whitepaper has the following to say about GT1.

3DMark03 whitepaper said:
Early in the design of 3DMark03, we knew from information in our benchmark databases that a significant portion of PCs had DirectX 7 graphics hardware. We needed one game test that could run on these mid-range PCs. For these, a lighter game test was required.
.......snip.............
This test is not meant as a definitive evaluation of DirectX 7. It is not designed to give the average performance of DirectX 7 3D graphics usage. Typical DirectX 7 games use fixed vertex processing, whereas this game test uses 1.1 vertex shaders. We believe this is the future of vertex processing on both graphics cards and CPUs.
The overall goal of game test 1 is to complete the collection of the four game tests as a test that can run on DirectX 7 hardware and one that requires a lower fill-rate. To fully evaluate DirectX 7 performance, the previous version of the benchmark, 3DMark2001 SE, is more appropriate.
IMO a satisfying answer.
 
So what does it test really ? 3 years in the future games ?

Actually, I would say it tests for "when games actually use pixel shading techniques in a global way." (Not just a patch of water here, a shiny car there, but being applied to the scene globally.)

In short, when will games like Doom3 arrive?

Ultimately, it's up to YOU to decide when you think that might happen. It's not 3D Mark's place to do that. What they are doing is creating an engine / benchmark which uses the features that the latest DX versions allow for. It's up to the game developers to decide when they want to do it, and the hardware vendors build cards that can handle it.
 
I would classify it as a graphics card benchmark rather than a gaming benchmark. I think it's more useful than 3Dmark2001 as I can identify what upgrades are most likely to have what effect.

ie. I have a P3-1Ghz CPU w/ 9700Pro. My 3DMark2003 score is 4019.

Practically every game I play I will set the max detail features on and get a good playable fps, for instance BF1942 50 - 85fps.
UT2003 is a little different, same max graphics settings BUT I cannot use EAX HD and I get the best fps improvement by reducing Karma.
If I had kept my TI4200 and upgrade the CPU x2.5 I still wouldn't be able to run the game with max settings.
3Dmark2003 clearly shows that I'm taking a big hit with Sound and CPU maths whereas 3Dmark2001 just showed an overall score.
By showing how a GPU effects the CPU you can better gauge how much to spend on CPU v graphics card.
 
Back
Top