Interesting, I find the situation a little ironic
- Thread starter ben6
- Start date
Going to play devil's advocate for a bit here...
I can agree that nVidia is being very hypocritical by dismissing 3DMark03 while praising 3DMark2001 at the same time. But what about those 3DMark2001 naysayers now praising 3DMark03? Aren't they being hypocritical as well? We have the previous 3DMark2001 naysayers getting on nVidia's case about flip-flopping their position since they liked 3DMark2001 but don't like 3DMark03 - yet, at the same time, those naysayers hated 3DMark2001 and now like 3DMark03 - making them guilty of the very thing they are accusing nVidia of, flip-flopping! If you can turn around and say 3DMark03 is okay while 3DMark2001 isn't, then why can't nVidia do the opposite? Having said that...
If nVidia's arguments against 3DMark03 hold no water because they can be applied to 3DMark2001, then what about the praises for 3DMark03? Can those same praises be applied to 3DMark2001? In that quote, you pretty much say that nVidia's position on 3DMark03 should be the same as their position on 3DMark2001 because 3DMark03 is still playing by the same rules, so-to-speak... However, before that:
Now here, you say that there was a change between 3DMark2001 and to 3DMark03 - that the rules have changed. You say 3DMark2001 was unbalanced, but now praise 3DMark03 - presumably because you feel its balanced. So I'm confused now. At one point you say that 3DMark changed from 2001 as an unbalanced benchmark to '03 as a balanced benchmark, therefore justifying your praise of 3DMark03. But then when nVidia dismisses 3DMark03, you say that they're aguments hold no water because what they say about 3DMark03 can be used against 2001 - implying there was no change between them.
You can't have both, which position do you take? Has 3DMark03 drastically changed from 2001, allowing you to give praise to 3DMark03 while dismissing 2001 and at the same time allowing nVidia to go against 3DMark03 while priasing 2001? Or has it not - declaring nVidia a cry-baby just because they lost and at the same time declaring your new position on 3DMark as nothing more than cheerleading reaction because nVidia is now losing in that benchmark?
martrox said:
I can agree that nVidia is being very hypocritical by dismissing 3DMark03 while praising 3DMark2001 at the same time. But what about those 3DMark2001 naysayers now praising 3DMark03? Aren't they being hypocritical as well? We have the previous 3DMark2001 naysayers getting on nVidia's case about flip-flopping their position since they liked 3DMark2001 but don't like 3DMark03 - yet, at the same time, those naysayers hated 3DMark2001 and now like 3DMark03 - making them guilty of the very thing they are accusing nVidia of, flip-flopping! If you can turn around and say 3DMark03 is okay while 3DMark2001 isn't, then why can't nVidia do the opposite? Having said that...
Doomtrooper said:
If nVidia's arguments against 3DMark03 hold no water because they can be applied to 3DMark2001, then what about the praises for 3DMark03? Can those same praises be applied to 3DMark2001? In that quote, you pretty much say that nVidia's position on 3DMark03 should be the same as their position on 3DMark2001 because 3DMark03 is still playing by the same rules, so-to-speak... However, before that:
Doomtrooper said:
Now here, you say that there was a change between 3DMark2001 and to 3DMark03 - that the rules have changed. You say 3DMark2001 was unbalanced, but now praise 3DMark03 - presumably because you feel its balanced. So I'm confused now. At one point you say that 3DMark changed from 2001 as an unbalanced benchmark to '03 as a balanced benchmark, therefore justifying your praise of 3DMark03. But then when nVidia dismisses 3DMark03, you say that they're aguments hold no water because what they say about 3DMark03 can be used against 2001 - implying there was no change between them.
You can't have both, which position do you take? Has 3DMark03 drastically changed from 2001, allowing you to give praise to 3DMark03 while dismissing 2001 and at the same time allowing nVidia to go against 3DMark03 while priasing 2001? Or has it not - declaring nVidia a cry-baby just because they lost and at the same time declaring your new position on 3DMark as nothing more than cheerleading reaction because nVidia is now losing in that benchmark?
What i found even more funny is that while Nvidia has something to gain by flip flopping (i haven't actually seen anywhere where Nvidia praises 3D Mark i might add, although i haven't found any complains either) their position, the former 3D Mark naysayers has nothing to gain. At least not money
Nice try, but there's a logical fallacy there. Saying Nvidia's arguments hold no water does not mean there is no change between the two, only that the arguments used do not speak to the change.
Anyways, here's a change that does make 3DMark03 more useful today than 3DMark01 was last week: it's a lot newer. 3DMark01 was supposed to more or less predict performance on a late 2002 game. And it did a pretty good job considering it was released 2 years ago. But if I want to see how a system performs running a late 2002 game, I can just look at benchmarks from late 2002 games, which are even more accurate than 3dMark01 at predicting their own in-game performance.
OTOH, if I want to approximate how a system will run a late 2004 game, I can either run 3DMark03--knowing full well that at best it will probably only give a pretty good approximation--or I can...wait until late 2004.
As to people who argued against 3dMark01 back in March '01 when it was released, then yeah, you have a pretty good point. But it's perfectly consistent to think 3DMark01 was useless to still be using a week ago but that 3DMark03 is the bee's knees.
Dave H said:
Right, I should have made myself more clear. Obviously, it's newer, and changes have been made to make it more up-to-date. But I was saying its more or less the same program in the sense that its objective is still the same - to predict how well a system will perform on future games. I don't think 3DMark changed its position at all, in that sense. The only significant change I see in the release of 3DMark03 is that nVidia did not come out on top this time, and I think that's the cause of all the flip-flopping.
And that's exactly who my post was directed to.
dksuiko,
I thought 3dmark 2001 was a bad benchmark when it came out, I still think it is a bad benchmark. I thought 3dmark03 would have the opportunity to be a much better benchmark before it came out, and I currently think it looks like it has achieved that, though I am welcome to a rational argument addressing why I am in error.
The thing is, I am not Doomtrooper, and in addressing his comments you have not addressed mine, yet I am still someone who fits the general description of the position you are criticizing. As an alternative to quoting my paragraphs of text associated with this myself in case it was only Doomtrooper you were disagreeing with, I ask you, as a continuation of your devil's advocacy, to do a search on my name and respond as you see fit if you disagree for the reasons I have given.
A short summary: if 3dmark03 was all still like its Game 1 test, I'd be saying it was exactly as poor a benchmark as 3dmark 2001 was, but as the paradigm for the way GPUs offer feature enhancement (computation using common basic functions instead of adding custom fixed function hardware) for games shifts, so does, in my view, the applicability of a benchmark's scalability.
And, please, if you disagree, pick one of my posts outlining the above for a starting point for reply.
I thought 3dmark 2001 was a bad benchmark when it came out, I still think it is a bad benchmark. I thought 3dmark03 would have the opportunity to be a much better benchmark before it came out, and I currently think it looks like it has achieved that, though I am welcome to a rational argument addressing why I am in error.
The thing is, I am not Doomtrooper, and in addressing his comments you have not addressed mine, yet I am still someone who fits the general description of the position you are criticizing. As an alternative to quoting my paragraphs of text associated with this myself in case it was only Doomtrooper you were disagreeing with, I ask you, as a continuation of your devil's advocacy, to do a search on my name and respond as you see fit if you disagree for the reasons I have given.
A short summary: if 3dmark03 was all still like its Game 1 test, I'd be saying it was exactly as poor a benchmark as 3dmark 2001 was, but as the paradigm for the way GPUs offer feature enhancement (computation using common basic functions instead of adding custom fixed function hardware) for games shifts, so does, in my view, the applicability of a benchmark's scalability.
And, please, if you disagree, pick one of my posts outlining the above for a starting point for reply.
dksuiko said:
In all the excitement, I'm sure it looks like hyporacy on both sides, HOWEVER..... I'm of the opinion that NO single bencmark should be the be-all/end-all of a product, and that applies to 3DMarkXXXX. BUT, 3DMark should taken into consideration. Also, there seems to have been a radical departure with this new 3DMark, as it seems to far more forward looking than in the past......AS IS DX9!!! AS a DX benchmark, it can only go as far as DX goes.......and 3DMark03 is much more DX9 oriented that 3DMark2001 was DX8 oriented. Here's a question..... When nVidia's product line as progressed to cover the market in NV3x solutions, will nVidia then embrace 3DMark? ANSWER: IF they do well in it.... of course they will!
martrox said:
I would agree with you that it's far more DX9 oriented then the 2001 version was DX8 oriented. But i still have issues with it with regards to DX9. I would have liked the DX9 test to be a lot more hard on the cards and at least one of the DX8 tests to be a bit faster. At least in the 10-15 fps range. And, as it is now, the DX9 cards are just as fast in GT 3 and 4. The GF FX is even a bit faster in GT4 (DX9).
As for Nvidias claims with regards to Doom3, well, it sure looks bad with regards to performance. (as in Doom3 probably a lot better). But what's more interesting is if the relative performance between the cards will stay the same. If it does, then the performance issues are a bit less of a concern. I still don't really like it though because no matter how much you tell people that it should be used only as a relative performance/ability indicator, many will still use it as a pure FPS benchmark. And that will probably be a bit misleading.
Anyone know the approximate scene poly count for Doom3? From screenshots it seems like it might be lower than in GT2 and GT3, which could explain much of the performance difference. Unfortunately, Futuremark doesn't give us the actual scene poly counts (rather the rendered poly counts), but assuming each poly undergoes an average of 2 passes on PS 1.4 (z-pass plus 1 light), they would be 75,000 for GT2 and 140,000 for GT3.
Joe DeFuria
Legend
Agreed. And it's the journalists responsibility to use the benchmark appropriately, rather than just throw out the 3DMark numbers on a pretty graph and say "look...this card is faster than the other card."
In any case, 3DMark03 is NO DIFFERENT than 3DMark 2001 in this respect. And yet, "all the sudden", some web-sites now have some "revelation" that 3DMark03 isnt accurately measuring "performance?"
This tells us exactly how clueless these journalists are. And rather than fess-up and say how they've been using it WRONGLY for the past 2 years, they've decided to blame the benchmark for being something they don't know how to use properly.
Nvidia can complain as much as they want about 3DMark03 not doing things the way 'real' games aka Doom 3 do things, but honestly, I don't think that actually matters. Why do I say this? Simple, the results from 3DMark03 actually seem to be fairly similar to the results of comparisons using Doom 3, even if the FPS is lower. GFFX > R9700 > R9500 > GF4 > R8500 > GF3 and so on.
I think the problem is the numbers are exaggerated in 3dMark03.
Joe DeFuria
Legend
FutureMark can't win.
Either their benchmark isn't "Exaggerated enough", 3DMark 2001, or it's "too exaggerated" '03.
I think one of the issues at this specific time, is that pixel shading is really a paradigm shift for GPUs. Most would agree with this: Hence' nVidia' "Dawn of cinematic rendering" campaign, etc.
We are so used to just judging cards based more or less on FILL RATE alone. Now, that shading has come into the mix, how do you rate a card's "shading" ability in combination with it's "fill-rate" ability?
If two cards are equally good at "DX7 style mutitexturing", but one also is able to do "pixel shading" at a much more reasonable rate (or at all)....how do they compare?
This is why the scores are "exaggerated" beyond mere "fill rate" ability. And it doesn't matter TOO much to what extent 3D Mark exaggerates it, because there will always be people who think it's too much, or not enough.
And they look "exaggerated" now, when we're showing DX7, DX8, and DX9 cards all at once. What happens in 18 months from now when it's mostly DX9 cards all together?
UT2K3 fly-by scores are also too inflated compared to actual difference in a game. Yet, they are still used (and abused for that matter). The sites that use Fly-By CORRECTLY, are the ones that explain it's not representative of actual difference in performance in the game, but more to show the difference in potential for the GPUs in comparison.
The solution, (as I keep repeating), is for journalists to use the benchmark properly in context.
Joe DeFuria said:
You're probably correct. But imo, they could have done better. And rather easily also as i see it. Change the 2'nd DX8 test so that you would get something like 10-20 fps on the fastest DX8 hardware and instead make the DX9 test tougher. As it is, you would think that even the R9700 won't be able to run any DX8 game (which i guess you could say that Doom3 is) with FSAA or aniso. It's getting close to unplayable numbers even without it.
Joe DeFuria said:
Ok, lets keep the benchmark to only being used properly, because we know thats easy.
The fact of the matter is it will be used improperly, regardless of how many times you post that it shouldn't be. OEMs will trumpet these numbers on the back of their boxes, and lay people in Best Buy will be "mislead" into thinking product A is 2x faster/better, when it isn't in real life, not now, and not in the reasonably forseeable future.
While I'm all for getting rid of shaderless cards, and Futuremark will accellerate that by dissing them hard, I don't think its a very good indication of future games for the general public.
Similar threads
