Sections that
HotHardware state are excerpts from the comments nVidia are circulating about 3dmark03:
nvidia said:
"3DMark03 combines custom artwork with a custom rendering engine that creates a set of demo scenes that, while pretty, have very little to do with actual games. It is much better termed a demo than a benchmark. The examples included in this report illustrate that 3DMark03 does not represent games, can never be used as a stand-in for games, and should not be used as a gamers’ benchmark."
Well, this just strikes me as hypocrisy, but I tend to still agree. What I think has changed compared to 3dmark 2001 is that all vendors are playing the shader performance game (or intend to), and benchmarks limited by shader functionality is a more predictable criteria for indicating future performance since the methods of exploiting it going forward will depend on the same fundamental benefit to a large degree (how fast basic shader instructions can be executed). The issue I see as far as that goes is whether the techniques used are optimal, or atleast reasonably optimal for a wide variety of cards...
nvidia said:
"Unfortunately, Futuremark chose a flight simulation scene for this test (game 1). This genre of games is not only a small fraction of the game market (approximately 1%), but utilizes a simplistic rendering style common to this genre. Further, the specific scene chosen is a high altitude flight simulation, which is indicative of only a small fraction of that 1%."
Well, it seems similar to space sims as well as flight sims to me, and that might grow the 1% figure a bit (where are the percents for the game types they would have preferred to put it into perspective, and the recognition of the weighting of this test in the score?). And what is the "game market"? Games of the type sold altogether, or 3D games? Is it percentage of game owners who play flight simulators (perhaps the number of actual importance for determining the applicability of the test), or the percentage of games sold that are flight sims? Is that 1% as "pulled out of a stinky place" as it sounds to me, or is it based on even one of the questionable, but real, determinations above?
Their argument strikes me as a red herring...if I were to attack its applicability as an indicator of game performance, I'd focus on whether the flight modelling was as demanding on the CPU as it would be in an actual flight simulator.
But as far as I'm concerned, 3dmarks have never been a good indication of game performance, and the blind adherence to it as such was not any more correct when nvidia profited from it. What I've hoped for it to be is a good indicator of graphics card power that the user could learn to more accurately use as part of an evaluation when comparing cards. The problem was, in my view, that the previous 3dmark (except synthetic tests) couldn't even serve that function well. In my view, due to shaders, the game tests have become more like "complex synthetic tests" than they could have been prior, and as such the results are more useful...but, then the actual fps values would serve too, or perhaps even better, using frame based rendering and determining time to completion. Hmm...I really like the idea of the last for the shader tests...keeps things in the proper proportion in my view (think back to the UT 2003 thread for my reasoning
). The added bonus of this is that the "mindless" 3dmark mentality might be circumvented a little bit.
nvidia said:
"For all intents and purposes game tests 2 and 3 are the same test. They use the same rendering paths and the same feature set. The sole difference in these tests appears to be the artwork. This fact alone raises some questions about breadth of game genres addressed by 3DMark03. --- These two tests attempt to duplicate the “Z-fi"rst†rendering style used in the upcoming first-person shooter game, “Doom 3â€. They have a “Doom-like†look, but use a bizarre rendering method that is far from Doom 3 or any other known game application."
I wish nvidia would provide the infor analyzing this. This "bizarre rendering method" could be a valid concern... or a complaint about PS 1.4 functionality and architectural failings with the GF FX with a heavy spin.
nvidia said:
"Finally, the choice of pixel shaders in game tests 2 and 3 is also odd. These tests use ps1.4 for all the pixel shaders in the scenes. Fallback versions of the pixel shaders are provided in ps1.1 for hardware that doesn’t support ps1.4. Conspicuously absent from these scenes, however, is any ps1.3 pixel shaders. Current DirectX 8.0 (DX8) games, such as Tiger Woods and Unreal Tournament 2003, all use ps1.1 and ps1.3 pixel shaders. Few, if any, are using ps1.4."
This strikes me as quite the tremendous smoke screen, with, again, a notable absence of logical consistency with the CineFX push. Also striking is the odd feeling of "shoe on the other foot"-itis in regards to ps 1.4, but I also still think it raises an important issue...
A simple question: can PS 1.3 reduce the number of passes for the techniques likely used in 3dmark, or in some other way enhance performance significantly over 1.1? I know Carmack's comments don't make me think so right now.
A not so simple question: could a HLSL compiler offer performance advantages for a "ps 1.1 compatible" shader compiled to a ps 1.3 target?
On this issue, I might end up being in complete agreement with nvidia, depending on the answers to these questions, especially the latter.
nvidia said:
"This year’s 3DMark has a new nature scene (game 4). It is intended to represent the new DirectX 9.0 (DX9) applications targeted for release this year. The key issue with this game scene is that it is barely DX9.
Heh, what constitutes "barely DX 9"? I'm suspecting it is using shader 2.0 functionality instead of "2.0+". Despite all sorts of hypocrisy and spin alarm bells going off, I'd be inclined to agree if there are opportunities for "2.0+" to improve functionality significantly. I certainly had similar feelings about the prior nature scene and the various issues of scoring (which I don't think is as much of an issue this time around, as we're on the cusp of shader enabled games appearing) and failure to be optimized for higher capability (which I feel could very well still apply, as I
strongly feel that this test at the very least should have been a HLSL showcase).
