Hyp-X said:
Is it doesn't reflect their concerns because they quit the beta testing,
- or -
they quit the beta testing because FM didn't what to "reflect their concerns".
A bit of both, perhaps. Frankly, given how late in the process Nvidia quit, I doubt 3dMark03 would have significantly different performance characteristics (on Nvidia hardware or otherwise) had they stayed in. But I don't have any details, so I don't know. Since Nvidia has been implying heavily that this is the case (recently going so far as to, in what is apparently going to be their only response to this whole scandal, say "we don't know what they did, but it looks like they have intentionally tried to create a scenario that makes our products look bad"), it bears mentioning that they only have themselves to blame for quitting.
Did they quit in the first place because their concerns weren't being met? Well, yes, they did--although obviously the concerns they now say are "intentionally creating a scenario that makes our products look bad" (the changes from build 320 to build 330) are not the concerns they had when they decided to quit in December. Between the
whine paper they sent to all the sites (except B3D of course), and the competitive landscape at the time 3dMark03 was released, I think it's relatively clear what Nvidia's concerns were that weren't being met. At the time 3dMark03 was released, Nvidia was trying to position its GF4Ti series against ATI's 9500/9700 series, and its GF4MX series against ATI's 9000/8500 series. 3dMark03, being a forward-oriented benchmark, put these comparisons in a pretty dim light.
As any genuinely forward-oriented benchmark would have.
As for Nvidia's specific complaints, I spent a lot of time analyzing them back then, and came to the conclusion that they were generally bunk, with the possible exception of calculating silhouette edges in GT2 and GT3 on-GPU with a VS 1.1 shader instead of on-CPU (because I didn't understand the trade-offs well enough to comment, and because the posters at opengl.org, who did, seemed to agree that this was suspicious). Even this complaint (and the related skinning complaint), however, has to answer to the fact that, despite Nvidia's explicit claims to the contrary,
GT2 and
GT3 are not by any stretch of the imagination vertex shader limited, on GF4 or any other card.
Here's one of my posts from back then discussing most of the issues (written in the context of Wavey's article on the subject). As you can see, I found a number of the choices FM made somewhat questionable, and generally would have come up with a somewhat different result if it were up to me to create a benchmark along their stated goals. Reading it over today, I'd say I agree pretty wholeheartedly with what I said back then. The only change I might make is that back then I was more on the stencil shadowing bandwagon whereas now I realize that near-future games are likely to be using techniques such as shadow maps as well. So while I did say back then that having two tests to measure the same rendering technique seemed like a bad idea, now I would probably add the suggestion that one use shadow maps instead.
Hyp-X said:
I said:
As for issue d, Futuremark obvious makes a conscious decision to use a general rendering path, based around the DX9 spec, rather than use vendor-specific rendering paths.
The fun thing is they have different rendering paths, but they say they are not vendor-specific paths.
Since they do this they have no basis to justify those paths.
I assume you're referring to the fact that GT2 and 3 have a PS 1.4 rendering path and a multipass PS 1.1 fallback? If so, a couple responses. First, both paths embody valid approaches from the point of view of the DX 8.1 spec, so they are in some way not vendor-specific but merely different-levels-of-the-spec-specific. Of course, it's obvious that FM would not have bothered to code the PS 1.1 fallback path if there were not hardware which could run PS 1.1 but not PS 1.4.
So does this completely demolish FM's claims to be hardware-neutral? Not really; it only demolishes the straw-man argument that they should be hardware-ignorant. Again, it all comes down to compromise. FM's attempts to be as fair as possible should not require it to ignore reality.
Making a benchmark like 3dMark requires a balance between fairness/vendor-neutrality and accuracy. Sometimes one side is compromised, sometimes the other. You seem to be saying that if FM cannot purely uphold one side or the other, the benchmark has no meaning or merit. That seems silly to me.
Hyp-X said:
So you agree that by disallowing equal quality optimizations (I mean the things ATI did, and NOT what nVidia did), they becoming less accurate at displaying future game parformance?
Cool
Sure, of course, particularly when it comes to high-profile games. When it comes to low-profile games, such a choice might make the benchmark more accurate. But giving up some accuracy for fairness does not mean that the benchmark has no accuracy whatsoever.
Hyp-X said:
I said:
Are you referring to me? I've certainly argued that 3dMark03 has some relevence in respresenting the performance characteristics of future games. I hope I've never claimed, though, that there will not be a systematic performance difference between it and those games, as if 3dMark03 is a 2004 PC game traveled back in time to today.
I didn't say it was possible to write such a benchmark.
But some people think it is.
Not neccessarily you.
I'm not sure some people really think this. I've never seen such a statement made by anyone too credible on any forum, nor by Futuremark themselves. I think you may be chasing a straw-man.
Hyp-X said:
The problem is not when you'll see future games give you different framerates (that's inevitable).
The problem is when you'll see future games having a different vendor bias.
And for this issue to be solved it's NOT the IHV's but the ISV's the right ones to consult with.
IHV's couldn't tell you what features will be used in future games, only maybe what they'd like to be used.
Agreed 100%. This fits in with something I noticed when I was looking at the membership of the SPEC Consortium for my earlier post: it's chock full of members who are actual users of scientific computing simulations, like the national labs and so on. FM's beta program now has representation from the IHVs and the press. ISV representation would be a big plus, and could probably improve the benchmark a great deal.
Hyp-X said:
Again I say if a mid-2004 PC game will run either 2x fast or 0.5x as fast on most of the video cards of that time than 3dmark, then it's good.
It's not the absolute, it's the relative performance that matters.
Agreed again. And I'm not at all sure that the attempted vender-neutrality of 3dMark03 doesn't produce some systematic performance bias as compared to actual results from the games it tries to model (as I wrote in my last reply). I do think that the fact that most games are written in DX, that DX is a relatively "narrow" API and each particular DX level "embodies" a particular level of hardware pretty closely, and that many games will resemble the 3dMark03 game tests in rendering style and workload, all mean that whatever systematic bias exists won't be so great as to make the results lose all predictive power.
Hyp-x said:
I said:
Do I seriously think 3dMark03 compromises its predictive abilities if FM refuses to allow special-case optimizations in IHV drivers because doing so fails to model this "last ditch, devrel didn't do enough to prevent a/b/c/d during game development" motivation for driver optimizations? No. Do you?
Well "devrel didn't do enough to prevent" is a one sided view.
What about hardware that came out half a year after the game?
What could they do?
They either ask the game developer to create a patch with a new optimization, or they try to do that in the driver.
FM will refuse the former, and then disallow the latter.
It'd be my guess that this issue is one in which the beta program works especially well. In fact I'd think this is one of the explicit purposes of having the IHVs play such a large role in the development of the benchmark: to make sure it the final benchmark will be generally optimized not just for their current products but for their upcoming ones as well. (And it's one of the reasons for confidentiality rules on the beta program.)
3dMark03 tries to capture the performance of the DX9 PS/VS 2 generation of cards. Presumably the IHV participation in the beta program allows them to do so fairly. When a truly new generation of cards comes out, FM should release a new 3dMark benchmark. (Whether PS/VS 3 will qualify or we'll have to wait for DX10, who knows? Personally I'd hope PS/VS 3 does get met with a 3dMark04.)
Hyp-X said:
So there are two game test with Doom3 like visuals, but with very different behaving codepaths than Doom3. Are other developers will do the same?
Which one of the tests represent techniques present in future RTS games?
Every developer will use stencil based shadows instead of shadow buffering?
What general techniques and rendering styles are we talking about?
As I said in my old post linked above, I don't think 3dMark03 got it perfect. Two stencil shadow game tests seems silly; again, one test with stencils and one with maps/buffers seems like a great idea. Maybe they should have used a HLSL for GT4 (although in hindsight this argument, common at the time, seems IMO a little silly now, part of a hoopla over HLSLs that is still slightly premature). And I'm not sure GT1 really has a clear purpose in life; more of a "let's lump in a test with a DX7 fallback as a sop to GF4MX owners with a test of a left-out genre like flight simulation, and toss in some vertex shaders for kicks" sort of thing. (As for your RTS suggestion, it's interesting and would certainly provide a different sort of workload, although traditionally RTSs don't tend to be particularly stressful on the graphics end...although a graphically advanced "close-in" RTS like a spiffied-up WC3 certainly might.)
Is the selection of game tests in 3dMark03 perfectly representative of a cross-section of 3d performance-intensive 2H04 PC games? No. Are each of the individual tests somewhat representative of what many individual games in that category will be like? I'd say yes.
In the end I think our ideas about what 3dMark03 is really like are pretty similar. We both recognize that a benchmark like that is subject to an inherent tension between modelling the performance of (non-vendor-neutral) games accurately but itself remaining vendor-neutral.
The difference between our viewpoints lies in what we conclude from the existence of this tension. You seem to think that since such a benchmark will inevitably sacrifice something in terms of accuracy, we can't treat it as having anything accurate or useful at all to say about future game performance. I think that's too stringent, and creates an ideological purity where none is needed. You probably think I'm too wishy-washy, always admitting the contradictions of the approach and the flaws of 3dMark03 as it actually is, but never giving up on the idea that we can get useful information out of it anyways.
I think that we're having an interesting discussion. But in the end (not yet, necessarily) we might have to agree to disagree.
