3DMark05 and certain websites
- Thread starter Reverend
- Start date
Just my 2 cents.
Looking at 3dMark scores, how did it predict the performance difference between the X800 and 6800 in playing HL2? Or did it actually manage to show the difference between the X800 and 6800 in Doom 3 or Farcry?
For all this time, 3dMark has served one useful purpose, and that is the individual game tests, have been rather indicative of what the video card's limitations can be. Eg. Game test 4 shows a hardware's vertex shader's limits. But increasingly, I find that 3dMark scores do not truly reflect a hardware's capability in rendering graphics in the different engines available.
Let's look at the different engines available to us. Offhand, I can name the Cry engine, Unreal engine, Doom 3 engine, Source engine, Lithetech(sp?) and possibly there are so many more out there. Each engine has its own graphical quirks and capabilities. As benchmarks continually show, different hardware performs differently with games built on different engines, but there is a common trend. If a certain hardware does well on an engine, most games on that same engine performs better on that said hardware.
Which is why I think 3dMark might have to change. Are there games out there which use the graphical engines 3dMark uses? I don't know. But we do know of games that use many of the popular engines like the Unreal engine. Perhaps ChrisRay's idea of FRAPS might be good. Take the fps of the showcase games for each engine, and give us an idea on each hardware's capability with each engine, and perhaps we can make more informed choices on buying our hardware, based on our game preferences and maybe leave 3dMark scores to those who play games that don't use the popular engines, like the Sims 2.
Looking at 3dMark scores, how did it predict the performance difference between the X800 and 6800 in playing HL2? Or did it actually manage to show the difference between the X800 and 6800 in Doom 3 or Farcry?
For all this time, 3dMark has served one useful purpose, and that is the individual game tests, have been rather indicative of what the video card's limitations can be. Eg. Game test 4 shows a hardware's vertex shader's limits. But increasingly, I find that 3dMark scores do not truly reflect a hardware's capability in rendering graphics in the different engines available.
Let's look at the different engines available to us. Offhand, I can name the Cry engine, Unreal engine, Doom 3 engine, Source engine, Lithetech(sp?) and possibly there are so many more out there. Each engine has its own graphical quirks and capabilities. As benchmarks continually show, different hardware performs differently with games built on different engines, but there is a common trend. If a certain hardware does well on an engine, most games on that same engine performs better on that said hardware.
Which is why I think 3dMark might have to change. Are there games out there which use the graphical engines 3dMark uses? I don't know. But we do know of games that use many of the popular engines like the Unreal engine. Perhaps ChrisRay's idea of FRAPS might be good. Take the fps of the showcase games for each engine, and give us an idea on each hardware's capability with each engine, and perhaps we can make more informed choices on buying our hardware, based on our game preferences and maybe leave 3dMark scores to those who play games that don't use the popular engines, like the Sims 2.
why are we talking about x800 and 6800 , it predicted the power of the r3x0 and fx series very well . It showed how the early dx 9 cards handled early dx 9 tittles . 3dmark2005 if made as well as 3dmark 2003 was should show how the x800 and 6800 handle future trends in dx 9 gaming . Sorry to say but i think that the driver cheating and fms lack of balls to stand up to it is going to invalidate it .
Look at engines you have 3 main games based on engines , unreal 2k3 /4 is not a dx 9 game its barely a dx 8 game as it uses few shaders .
Doom3 is an opengl game but 3dmark did show the radeons problems with stencil shadows .
Farcry shows the r3x0 being supior to the fx line , just look at the fact that the radeons run in a higher shader lvl and faster than the fx line .
The source engine shows that the radeons are much faster than the fx line and the fx line is not good for pixel shading or vertex shading.
For all those who say that 3dmark 2003 is usesless are wrong. Yes the final score is useless expect comparing which card is faster at 3dmark. But the tests themselves show many interesting things
Unknown Soldier
Veteran
ChrisRay said:
Sorry ChrisRay .. I've seemed to have missed the other synthetic tests that you say can prove better than 3DMark03. Please expand on your answer so that I can also knwo what tests are actually better.
While it might not affect your buying decision of which graphics card you intend to buy in the future, it does affect alot of other peoples buying decisions. I've seen people buy GC's just because the card did alot better than it's competitor.
Unfortunately, people also get duped into buy a worthless graphics card because of the cheating that a certain IHV did/does. How many people have I seen buying a 5200 card just because it could do DX9(stated so on the box) and on the website.
And please don't say they deserved it then. No one deserves to be lied like that.
US
Unknown Soldier, I'm sorry, I was thinking of shadermark, Rightmark3d, and synthetic tests of that nature.
If you read above you will see why I prefer these applications, And why I think are better tools for determining performance.
Chris
This is a whole different issue though IMO, 3dmark holds alot of weight, But unfortunately that falls into the category of how people choose to use it as a tool. And it unfortunate that people base their buying decisions on products such as 3dmark. Heck I find it unfortunate people base their buying decision off of Doom 3.
But yes, That falls into the category of. Its not the tool, Its how people use the tool. Hopefully, people will study what they are buying. (However that does seem unlikely)
Chris
If you read above you will see why I prefer these applications, And why I think are better tools for determining performance.
Chris
This is a whole different issue though IMO, 3dmark holds alot of weight, But unfortunately that falls into the category of how people choose to use it as a tool. And it unfortunate that people base their buying decisions on products such as 3dmark. Heck I find it unfortunate people base their buying decision off of Doom 3.
But yes, That falls into the category of. Its not the tool, Its how people use the tool. Hopefully, people will study what they are buying. (However that does seem unlikely)
Chris
ChrisRay said:
Well, I don't understand this. What's different about 3DMark's rendering techniques?
We have per-pixel shading with normalmaps, skinning and stencilshadows. Isn't that exactly what Doom3 does aswell?
How does 3DMark03 not come close? You get a preset run, and you can run it as many times as you like, and take the average.
I asked you why it's synthetic, why did you not answer that?
I didnt feel the need to define synthetic for you. Because at such a point it would become an argument of semantics, Synthetic to me is anything thats not a real game enviroment. And thats exactly what 3dmark is. It's not a game enviroment, and I dont find much use in the test bed.
Well theres no AI, its one scene, Which is rendered the exact same way over and over again, Are you going to tell me Every Unreal Tournament map performs the same? or TR AoD, Far Cry, Half Life 2? You may be satisfied with the ease of using 3dmark03, That doesnt make the information as useful as other tests, I would much rather see a review which focuses on Gameplay, (In a real game enviroment) And the synthetics which handle individual synthetic tests.
Not really, If it were, There wouldnt have been the huge discussion on vertex skinning verses CPU skinning ect, dont believe the methods in 3dmark have accurately predicted Doom 3 or Far Cry, or HL2, Other than a generic fashion which I already mentioned.
And they certainly didnt tell us something we couldnt have figured out without looking at fill rate tester, Rightmark, or Shadermark, Tools far better suited for testing the specific functions you listed.
You dont need Game Test 4 to tell you that the X800 is superior in Vertex Shading to the Geforce 6800 series do you? You could have found that out in any other synthetic test which tests vertex performance. I usually give 3dmark one benefit of the doubt, "Its usually the first". But this time of round with shadermark and rightmark out there, Working on adding 3.0/2.0b functions to the software, I dont even believe that holds merit anymore.
In Any Event, I'm not gonna continue arguing a circular argument with you. We obviously disagree on the nature of synthetic tests, And we havent gotten anywhere in almost 2 pages. ;p
Chris
Bouncing Zabaglione Bros.
Legend
ChrisRay said:
That pretty much describes every real-life game's benchmark mode, and yet these games are somehow being held up as being less synthetic and more relevent than 3DMark's measuring stick, when in fact they suffer from all the same problems and limitations.
I don't see how anyone can criticise 3DMark's way of running benchmarks, without also criticising the way that every real-life game and app does it too.
ChrisRay said:
When I specifically ask you, apparently there is a need.
And you still haven't gone into specifics... "It's not a game environment", what exactly is a game environment in your view then? 3DMark03 has rooms, in those rooms you have characters. The characters are moving around, with skinned animation calculated in realtime, there are even physics applied. I may be missing something here, but it looks exactly the same as what I see in Doom3, or Unreal, or whatever other FPS game.
So if I'm missing something, please tell me what, and what its significance is.
I don't think AI has any significant effect on game performance in most cases. There may be exceptions in games with large numbers of enemies, but I suppose it's pretty obvious that your GPU has no effect on this. 3DMark03 mainly tests the GPU's capability of running games. If 3DMark03 says your GPU performs well, and your heavy AI game doesn't run fast, you can simply conclude that you need to upgrade your CPU.
Besides, this may come as a shock to you, but the key to having comparable benchmark results is to have results from exactly the same set of operations. In other words, benchmark results are useless if they are NOT rendered the same every time.
That is a useless point. Different games will apply different techniques. Doom3 will not predict anything about the performance of your system in HL2, because it's simply using different techniques. On the other hand, if Doom3 runs well, HL2 will most probably also run well, because they have pretty much the same system requirements.
However, any game using the same techniques as Doom3, will perform in the same fashion.
This obviously goes for 3DMark03 aswell. It cannot predict anything about games that work differently. But it is similar to most DX9 games, and therefore it is a good yardstick for DX9 game performance in general.
I disagree. You can test z-only fillrate, you can test stencilrate, and you can test shader performance. But these separate results won't mean a whole lot when you start combining the techniques. Certain combinations of techniques will work faster than separate, others will work worse than separate, because of special cases.
And it's hard to predict how these raw results would relate to actual game performance. Which one has the most effect on performance?
The best way to test actual game performance is to actually render a game situation. And this is what 3DMark does, and these other tests don't.
Why not run a real game then? That has been said many times before: the real games don't exist at the time of 3DMark's release. Unlike games, every issue of 3DMark is aimed mainly at one generation of hardware, and is released shortly after this generation of hardware, often years before actual games start using features of this generation of hardware.
That is nonsense. With the same logic I could say that you can find this out with 3DMark, so you don't need any other synthetic tests.
Also, 3DMark always has representative content, which most other benchmarks don't (they either render only simple triangles, or they use an old ATi Radeon 8500 demo with modified shaders or such, which is not representative for the workload that these cards can and should handle at all).
That's mainly because you fail to back your statements up with facts and logic. You remain stuck on your opinions, but don't give the facts and logic that these opinions are based on, assuming there are any.
Sure you can. I never said you couldnt. But the Tests in shadermark, Rightmark, Ect test specific function. But We've already covered that havent we. Testing Each individual shader functions, Shows us where the weakness of said hardware may lay. My criticism of 3dmark is that its results are generic, And less useful compared to a a test such as shadermark. When I am looking to determine shader performance. I look for situations I can see its near maximum throughput. This seems to be an opinion are unwilling to accept, But you dont have too.
But I already have Scali, You obviously dont like my answers, So maybe you should avoid asking the question, The reason I havent answered half of your questions is because you're asking the same thing over and over again, And repeating myself gets very tiresome. I feel have illustrated my points to why I dont believe 3dmark is as useful tool as say shadermark or rightmark,
Do you spend a lot of time playing 3dmark? Do you turn one corner, Then Reverse and turn another? If you do, Then maybe 3dmark is for you. But I do find it pretty odd that you seem to think 3dmark03 is actually like playing game when you run it.
Scali, There is a significant difference between playing Doom 3 and running 3dmark. Same with Far Cry or heck even the original Unreal, The various scenes in game perform differently, Render differently, If you think running through a scripted enviroment (Ala 3dmark) Accurately portrays gaming performance then that is your perrogative. I dont think they do at all, Due to the issues I have mentioned here.
I dont see how its nonsense anyway. Doom 3 you wont be playing at acceptable frame rates with a Geforce 6800 Ultra and a P4 1.0 Ghz, You've seen the thread here Scali, You know what I'm talking about, We both know why Game Test 2 is different from Doom 3 in CPU skinning, If its a useless point you wouldnt bother bringing up GT2 as an indication of Doom 3. I have never argued that 3dmark was entirely useless here, I have noted why I think it is less useful than the programs I mentioned above. I just believe there are much better tools out there. Who knows, 3dmark05 may changed my mind. I will have to test the program myself to draw any real conclusions about it.
And to your last point. This is why you make preset runs, And do multiple ones, You can easily reduce margin of error that way. If you are unsatisfied with the precision of the test runs, You can easily take more. The Average Minimum, Standard, And high FPS remains pretty constant when you do this.
http://dictionary.reference.com/search?q=synthetic
http://dictionary.reference.com/search?q=real
There are no semantics involved when I use those two words, They mean exactly what they are defined as in the dictionary.
I am done with this discussion Scali, (at least with you) We have been discussing this for 2 pages now, and there is no point continuing it. Perhaps others can have the oppurtunity to add their 2 cents in as well. If 3dmark suits your needs, Then please use it, Enjoy it on a level that it satisfies you. I on the other hand will look for other data when comparing video card performance.
Can't you just admit that you just don't like 3dmark Chris? It sounds like what you're trying to say without coming right out and saying it.
digitalwanderer said:
I thought his opinion was well thought out and intelligently delivered. Not full of exasperated emotion like many others around here. So what if he doesn't like it - at least he can clearly enumerate the reasons behind his thinking. Not that he needed me to defend his position of course
I'm just hearing his explanations of his dislikes and it just seems to me that he is looking more for reasons to dislike it than for ways to utilize it as a tool.trinibwoy said:I thought his opinion was well thought out and intelligently delivered. Not full of exasperated emotion like many others around here. So what if he doesn't like it - at least he can clearly enumerate the reasons behind his thinking. Not that he needed me to defend his position of course
digitalwanderer said:I'm just hearing his explanations of his dislikes and it just seems to me that he is looking more for reasons to dislike it than for ways to utilize it as a tool.
You're right but I read it differently. I didn't read that 3dmark03 is 'bad'. What I got (though I may be wrong) was that he feels that newer approaches that target specific capabilities like shader and vertex performance using tools like Shadermark and Rightmark are more useful indicators of graphics performance. 3dmark03 is just one particular application of this power based on Futuremark's crystal ball for future game designs whereas these tools give more detailed insight into the strengths and weaknesses of each product.
If I got it right you're saying:ChrisRay said:
(1) For learning about current game engine performance let's bench the current game engines (and nobody disagrees with you here).
(2) For benching potential future speed let's bench ShaderMark and similar tools, which bench some specific shader performance.
Did I get your opinion right?
Scali doesn't disagree with that, as far as I understand it. But there's one additional point which you seem to miss. Namely this one:
ShaderMark and similar tools don't even try to recreate a game like environment. They are not only synthetic, but also highly theoretical. They test some very specialized cases only. That's fine, they do have their purpose. But they don't tell the whole picture. One example: Card A and B might be equally fast while calculating shader XYZ. But when combining exactly this shader with some other things like e.g. multi layered textures or normal maps Card A might suddenly be 5x as fast as card B. That's why tools like ShaderMark are good, but don't paint the complete picture.
3DMark surely can't predict how much FPS we'll get in Unreal 2006. But the 3DMark guys *try* to create an environment (a collection of techniques) which come close to what future games are probably going to use. Sure, you can't really "play" 3DMark, and you can't freely move (at least not in the free version). But nevertheless 3DMark is a lot nearer to a real game than tools like ShaderMark. Not in the sense that you can "play" 3DMark, but in the sense that the 3D workload (amount + type + structure etc) is similar. 3DMark surely doesn't paint the whole picture, either. But it is another very interesting and important piece of the puzzle, when trying to figure out how future game engines might perform, which you seem to totally neglect.
ChrisRay said:
I'm just pointing out that 3DMark gives a slightly different angle on performance than the tests you mention. I haven't said I am unwilling to accept your opinion or anything.
I do think however that you are unwilling to accept that 3DMark is different from Shadermark or Rightmark or such tests. Or at least, you are not interested in these differences, and use this as an invalid reason to downplay 3DMark's usefulness in general.
You may prefer other tools, but that does not mean that those are the only tools people should use, or that others don't have any value.
I must have missed the answers to the specific questions I asked. Namely what makes 3DMark's game test different from say a timedemo in Doom3.
You only pointed out that Doom3 uses a slightly different method for rendering certain parts. But that's nonsense. The same goes for any other game that you might benchmark with. So I have to repeat my question because you haven't given an actual answer.
So well, if you do have an actual answer and I've missed it, please point out where you said it (what thread, what post, quote it if you like)... and if you don't have an actual answer, just say this is your opinion, and you cannot explain it to me, it's just what you think.
Perhaps it's because I'm a programmer, and actually have an idea of what the code for 3DMark03 looks like, and actually know that this is not very different from a game, even though they don't allow user input.
You may not understand this, but 3DMark03 can easily allow user input, and the special development versions that are available to beta members actually do. So yes, you CAN turn one corner, then reverse and turn another. So yes, 3DMark03 actually IS the same kind of interactive 3d engine as used in games.
Knowing this, I don't see a significant difference between running a 3DMark03 benchmark and running a Doom3 timedemo.
Perhaps for that very same reason 3DMark does not contain 1 game test, but several, each with a different type of environment, and different rendering features? Again, I don't see the difference.
As I say, you can do that with 3DMark and pretty much any other benchmark.
It's not what the words mean, it's why you are using these words.
WHAT is synthetic about 3DMark? What makes it different from a timedemo in whatever game? That's what I want answered. And that's where you failed to answer so far, afaik. So no, you're not done with this discussion as far as I'm concerned, because you failed in the basics of discussion, namely in getting your point across to the other side.
I don't even know what you mean exactly, so I can't even make up my mind whether I agree or not.
I was going to ask people to stick to the thread topic but after re-reading the 3rd page again, I've come to the conclusion that the thread is, in essence, still on topic. 3DMark seems to split people into one of three groups: (1) the "don't care" lot, (2) the "3DMark isn't synthetic enough and not a game either so it's useless" lot and (3) the "3DMark rendering, level design, etc are indicative of what near-future game engines might be like and therefore 3DMark is worth using" group.
I've always been a member of the 3rd group (although I'm usually labelled as being completely biased in this regard); sure there are plenty of elements I would like to change and I voiced some of them for 05, but I find the package to be as equally as useful as RightMark / ShaderMark / game timedemo. The key to 3DMark is, as with any benchmark, correct usage and interpretation of the findings. The PS1.1 and 1.4 shader tests in RightMark tell me nothing as to how well a graphics card will cope with per-pixel lighting in a modern game because it will most likely be used with other elements (such as stencil shadowing) that may, or may not, have a more significant factor in the overal performance; conversely the game tests in 3DMark do not highlight specific areas where a graphics card is strong or weak.
A review/preview of a graphics card should ideally have a batter of tests that expose the test product to an increasingly wider range of conditions and scenarios - starting with the totally "theoretical" such as Marko's fill rate tester, to restricted scenarios provided by RightMark and ShaderMark, to full gaming environments found in the likes of Far Cry and yes, 3DMark.
I've always been a member of the 3rd group (although I'm usually labelled as being completely biased in this regard); sure there are plenty of elements I would like to change and I voiced some of them for 05, but I find the package to be as equally as useful as RightMark / ShaderMark / game timedemo. The key to 3DMark is, as with any benchmark, correct usage and interpretation of the findings. The PS1.1 and 1.4 shader tests in RightMark tell me nothing as to how well a graphics card will cope with per-pixel lighting in a modern game because it will most likely be used with other elements (such as stencil shadowing) that may, or may not, have a more significant factor in the overal performance; conversely the game tests in 3DMark do not highlight specific areas where a graphics card is strong or weak.
A review/preview of a graphics card should ideally have a batter of tests that expose the test product to an increasingly wider range of conditions and scenarios - starting with the totally "theoretical" such as Marko's fill rate tester, to restricted scenarios provided by RightMark and ShaderMark, to full gaming environments found in the likes of Far Cry and yes, 3DMark.
I believe I'm in the same category (3) as Neeyik.
I also agree that a review should cover as many areas as possible.
Although personally I'm not all that interested in raw numbers, because these days the hardware has many optimizations and special cases (such as hierarchical z/stencil buffers), so testing a single feature is generally not at all representative for using that feature in a real-world scenario.
Basically the hardware is now designed to be optimal for games, and not just perform a single trick as quickly as possible. Therefore only testing a single trick is just not representative anymore in most cases.
I also agree that a review should cover as many areas as possible.
Although personally I'm not all that interested in raw numbers, because these days the hardware has many optimizations and special cases (such as hierarchical z/stencil buffers), so testing a single feature is generally not at all representative for using that feature in a real-world scenario.
Basically the hardware is now designed to be optimal for games, and not just perform a single trick as quickly as possible. Therefore only testing a single trick is just not representative anymore in most cases.
Oddly enough, it seems to me the members of Neeyik's Group 2 tend to be somewhat NVidia-centric. Perhaps this is because they tend to agree with NV's previous denigration of 3DMark after all the trouble regarding the 'optimisations'?
Personally, I'm in Group 3 - how can having yet another good benchmarking tool possibly hurt? If the 3DMark scores don't tally with other synthetics or timedemos then they could be doubted but as previously mentioned, the results have been pretty much spot on so far.
I'm expecting NV40 and R420 to be pretty much on a par in the overall 3DMark2005 score (as they are in games, Doom3 being the exception). If one outperforms the other greatly, I hope we won't be encountering the same FUD/driver cheats which we've seen previously.
Personally, I'm in Group 3 - how can having yet another good benchmarking tool possibly hurt? If the 3DMark scores don't tally with other synthetics or timedemos then they could be doubted but as previously mentioned, the results have been pretty much spot on so far.
I'm expecting NV40 and R420 to be pretty much on a par in the overall 3DMark2005 score (as they are in games, Doom3 being the exception). If one outperforms the other greatly, I hope we won't be encountering the same FUD/driver cheats which we've seen previously.
Similar threads
