Interesting (3DMark2003) article at Aces Hardware

[sonix666]

Hey guys,

the guys at Aces Hardware have come across some very interesting results concerning 3DMark2003. It seems that three of the four gaming tests of this benchmark are so GPU limited that a PII 350 MHz with Radeon 9700 can keep up with a P4 2.8 GHz with a Radeon 9600. However, the PII can't keep up in any of the real world gaming tests.

http://www.aceshardware.com/read.jsp?id=60000242

A quote from the conclusion:

For a synthetic graphics accelerator benchmark, this is an ideal situation. But this is not how 3DMark03 has been represented. It is stated to provide an overview of your "system's current gaming performance." Yet from our results, it has failed in that charge twice, by grossly exaggerating the performance of a 350 MHz Pentium II with a Radeon 9700 Pro and minimizing the performance of a 2.8 GHz Pentium 4 with a Radeon 9600.

Personally I think this is a confirmation that 3DMark2003 is severely overdone as what is supposed to be a gamers benchmark.

 

[AAlcHemY]

3Dm 2k3 is a 90% gpu benchmark, those results dont suprise me. 2k1 is mutch more balanced imo.

 

[CosmoKramer]

How many of you are aware that 3dMark2001 was criticized because it was too much of a system benchmark

I think 3dMark03 is great in that it very much isolates VGA performance.

 

[OpenGL guy]

3Dm 2k3 is a 90% gpu benchmark, those results dont suprise me. 2k1 is mutch more balanced imo.

3D Mark 2001 was much more balanced, but GPU speeds have long since overtaken it. This is the reason why 6 out of 7 of the tests that count toward the score are just about completely CPU limited on today's (and last year's) high-end cards.

To put it bluntly, I went from 12000 3D Marks to 16200 3D Marks increasing my CPU speed from a XP1900+ to a P4 3.0 Ghz using the same Radeon 9700 Pro. That's an increase of 33%! When I plugged in my AIW 9800 Pro, my score went up to 17400. That's an increase of about 7%, yet the 9800 Pro is more like 15% faster than the 9700 Pro.

 

[Dave Baumann]

I think we all knew older games were CPU intensive, however ask yourself if this will be as much the case for newer titles. Look at the difference a high end card has in comparison to a mainstream board in a relatively new title such as Splinter Cell - this is going to even more evident when we see some more PS2.0 games bencharks (soon).

However, for the fact that the 3D benchmark is stressing the 3D element is "Yay" as far as I'm concerned.

 

[sonix666]

Dave,

games now and in the past have been a balance between CPU and GPU power. Some games rely more on CPU power, others more on GPU power. I don't foresee this changing much in the near to far future. Sure, the 3D part might be more stressfull for your GPU, but at the same time things like AI, gravity, etc. will get more advanced and will stress the CPU at the same time. So, I don't foresee a future where games are purely GPU bound.

And sorry to say, but Aces benchmarking shows that 3DMark2003 is NOT an indicator at all for games now, the past and I don't foresee the future. And that is completely the opposite of what they state in their description of 3DMark2003. So, their benchmark description is misleading.

 

[Scott C]

Dave, although you are right that newer games will push the limits of the GPU and especially older video cards, they certainly won't be decreasing the CPU work they do such that you could downgrade your CPU without a problem.

EDIT:

Put another way, 3DMark 2k3 may have the "future games will stress your GPU like this" part down and somewhat realistic, but they definitely don't have the "future games will stress your CPU like this" part down at ALL.

Future games may require a faster, dX9 class GPU for the best performance, just like 3dMark 2k3 indicates, but they most certainly will need more than a 350Mhz CPU, unlike what 3dMark 2k3 indicates.

 

[Dave Baumann]

games now and in the past have been a balance between CPU and GPU power. Some games rely more on CPU power, others more on GPU power. I don't foresee this changing much in the near to far future. Sure, the 3D part might be more stressfull for your GPU, but at the same time things like AI, gravity, etc. will get more advanced and will stress the CPU at the same time. So, I don't foresee a future where games are purely GPU bound.

3DMark uses the exact same physics engine that many titles already use, and many more are likely to use in the future as well.

And sorry to say, but Aces benchmarking shows that 3DMark2003 is NOT an indicator at all for games now, the past and I don't foresee the future. And that is completely the opposite of what they state in their description of 3DMark2003. So, their benchmark description is misleading.

Two of the titles Ace's use are known to be horrifically CPU intendive. JKII, for instance, doesn't even use hardware geomtry processing - is this indicative of current and upcoming titles? UT2003's represents a small surprise, but then it is more of a DX7 title than DX8. Throw in some decently shader intensive titles into that mix and you'd probably get quite different results.

However, as far as I'm concerned as a reviewer for a 3D specific site if an application is a 3D graphics benchmark and its actually stressing 3D graphics then thats perfect! There are, afterall, plenty of games out there to represent game playing performance!

 

[Dave Baumann]

Dave, although you are right that newer games will push the limits of the GPU and especially older video cards, they certainly won't be decreasing the CPU work they do such that you could downgrade your CPU without a problem.

Who's suggesting that? Of course CPU's will stil remain a heavy element in a games performance, but more of the 3D rendering performance will become reliant (bottlenecked) on the 3D hardware as shaders become the base target.

That said, though, the role of the CPU will gradually further and further diminish in terms of the 3D rendering as time goes on. As I said, JKII relies on the CPU to do all the geomtry calculation, and that was the norm until not long ago - but now hardware geometry processing is the norm, thus reducing the overhead on the CPU for static environments to start off with. As vertex shaders are used, more of the character animations will be done in hardware, further reducing the work the CPU carries out. We've pointed out that system performance is important becuase of the geomtry data thats shuttled about by the system to the graphics board - but as Vertex Buffering is used more that will be stored on the 3D board, thus reducing the system overhead for rendering further.

This is the point of 3DMark - its a little more visionary. They are already employing a number of the elements I talk about there, but current titles don't - titles in the future will over a period of time.

Just because games that you see now don't seem to bear much relation to 3DMark now, doesn't mean they wont in the future. In fact, you could quite easily compare the types of things employed in games today to the rendering techniques seen in 3DMark2001.

 

[Althornin]

as games in the future move more geometry processing onto the video card, 3dmark03 results will beomce more and more relevant.
It is "FUTUREMARK" - who needs "TODAYSGAMESMARK" - we've already got the games!

 

[Slides]

Which future games are those? Any ones that are currently on the horizon? The games I currently play are certainly CPU bound.

 

[Sazar]

I just have 2 questions...

first of all... how much dx9 'stuff' is in gunmetal ?

I really would appreciate a leymans answer on this one because AFAIK... gunmetal qualifies as dx9 due to one shader... and even then just barely... I may be wrong which is why I want a clarification if it can be provided...

cheers...

secondly... did ace's 'reviewer' bother to read futuremark's whitepaper which specifically states it is a 3d test... and not a system test ?

it is really irksome to read these statements that are so completely out of line with what is written in the accompanying whitepaper of the product being 'reviewed'... erroneous/misleading statements like those in the article @ ace's are what is wrong with many online reviewers out there today... unfortunately...

 

[Slides]

So would this also be a misleading statement from FM since it is not exactly true?

The high quality game tests, image quality tests, sound tests and others give you an extremely accurate overview of your system’s current gaming performance.

http://www.futuremark.com/products/3dmark03/

 

[Sazar]

So would this also be a misleading statement from FM since it is not exactly true?

The high quality game tests, image quality tests, sound tests and others give you an extremely accurate overview of your system’s current gaming performance.

http://www.futuremark.com/products/3dmark03/

they are not used to calculate the score are they ?

please stick to relevant topic... the scores you garner are dependent on tests 1 thru 4... not on the other tests which give you an indication of your systems performance in various other tests...

 

[Dave Baumann]

first of all... how much dx9 'stuff' is in gunmetal ?

I really would appreciate a leymans answer on this one because AFAIK... gunmetal qualifies as dx9 due to one shader... and even then just barely... I may be wrong which is why I want a clarification if it can be provided...

It has one Vertex Shader 2 element (which can be run on the VPU), but supports on PS1.1 Pixel shaders. To my mind this is more of a DX8 benchmark than DX9.

 

[Slides]

The game tests are not part of the score?

 

[Sazar]

As with previous versions, 3DMark03 scores will initially range between a 1000 and 5000 3DMarks. They are
scaled such that an entry-level system will score approximately 1000 and a high-end system, at the time of
3DMark03 product release, will score approximately 5000. An entry-level system is approximately one with a
DirectX 8 graphics card and a CPU corresponding to 1 GHz. A high-end system roughly has a DirectX 9
graphics card and a CPU corresponding to 3 GHz. Of course, with time, the high scores are expected to keep
increasing. For example, for 3DMark2001 the best score immediately after the launch was close 5000, and
today’s high scores are over 20,000 3DMarks.

Each of four game tests generates a frame-rate (in frames per second) that is used to calculate the overall
score. The formula for calculating the overall 3DMark03 score is:
3DMark03 score = (Game Test 1 frame-rate x 7.3) +
(Game Test 2 frame-rate x 37) +
(Game Test 3 frame-rate x 47.1) +
(Game Test 4 frame-rate x 38.7)

This formula has been obtained by running the benchmark on a number of high-end systems. The results for
each system are weighted such that game test 4 gives 20% of the total score and game tests 1, 2 and 3 are
equally weighted to give the remaining 80%. Hence, game test 1, 2 and 3 each give 26.6% of the total score.
The weights are averaged across all the systems used to generate the weights in the formula. Note that this
weighting can be very different on low-end systems.
In 3DMark03, we have separated the measuring of CPU performance into a separate test and score. This score
has a different range than the 3DMark score, in order to prevent confusion between these two total scores. It is
balanced to give a CPU Score of 500 to high-end systems at launch time. The formula for calculating the CPU
score is:

CPU score = (CPU Test 1 frame-rate x 4.6) +
(CPU Test 2 frame-rate x 27.5)

This formula is obtained using the same approach, with the two game tests weighted so that each contributes
50% to the final score.
The table below shows sample scores on three PC configurations. Two systems have the same graphics card,
and two have the same CPU. Note that the CPU score rises significantly for the more powerful CPU. The overall
3DMark score is less affected by the CPU, but depends more on the graphics card.



Conclusion

Over the last four years, 3DMark has become a trusted standard in 3D graphics benchmarking. Futuremark’s
latest version, 3DMark03, continues the tradition of providing an easy-to-use tool for benchmarking nextgeneration
3D graphics technology. This time, powered by the latest DirectX 9 features and supporting graphics
hardware, we showcase 3D graphics likely to be seen over the next year and a half. More importantly we provide
a neutral and transparent benchmarking tool to allow the user to evaluate the capabilities of the latest hardware.
We have added new functionality, such as the 3D sound test, a set of new feature tests and much-improved
image quality tools, to allow the user to isolate performance of key parts of 3D graphics usage. As graphics
hardware becomes more powerful, especially with new sophisticated shader support, stunning 3D graphics will
become accessible to more applications. We believe 3DMark03 will serve as a highly dependable tool for the
benchmarking professional in this new environment.

some pics and tables I didn't copy since they were coming out garbled...

 

[Slides]

That's great. But before you accuse Ace's of misleading people, you should take a second look at FM's own statements wrt 3DMark03. Most people who use 3DMark will not read their white paper or whatever it is. They'll run it and compare the scores with others, and use it as a measurement of their system's gaming performance.

 

[Sazar]

That's great. But before you accuse Ace's of misleading people, you should take a second look at FM's own statements wrt 3DMark03. Most people who use 3DMark will not read their white paper or whatever it is. They'll run it and compare the scores with others, and use it as a measurement of their system's gaming performance.

yes... but ace's is not any old user or most people... they have some obligation to research the product they are supposedly reviewing and erroneously stating does something that its white paper claims otherwise don't you think ?

that is where the problem arises IMO...

ace's has taken the liberty to extremely selectively pick a choice sentence w/o any regards to the context or the rest of the white paper... this is where I feel they didnt do justice...

 

[Slides]

So you want Ace's to ignore what FM says on their main page and take into account some hidden white paper?

But for the specific issue of whether they looked at this whitepaper, you should take it up with them I guess. I haven't read this whitepaper so I don't know.