3DMark05 and certain websites

[Scali]

I think graphics technology just soared straight over the heads of most journalists in the past few years. Back in the VooDoo days, everyone still knew how things worked... All accelerators were very simple, and could only draw textured triangles. Whichever card was fastest, was the best.

But these days, I don't think anyone but the actual people who design the hardware, or develop software for it, actually understand what these massively complicated devices are really all about.
So I suppose the journalists have no choice but to parrot everything that NV, ATi or whoever else tells them.

It's not just 3d cards either... I've seen the same happening with CPUs. I recall that when the P4 was introduced, nearly all websites mentioned something like "Because of the improved branch predictor, the floating-point performance is increased".
Since nearly all websites mentioned this, I suppose Intel marketing told them this. But ofcourse it's complete nonsense this way. Either Intel or the websites pulled it out of context, but apparently nobody realized what they were saying anyway.

 

[ChrisRay]

I would still rather tests such shadermark be used to test performance, The FX wasnt slow in Every DirectX 9.0 game either, I'm not really interested in getting in an argument about 3dmark, But I will promptly ignore any scores, In future products with tests like Aquamark and 3dmark03/05, There are other good utilities for testing performance of such titles, The only real benefit I could see to using such tools is to get more content into a review quick because they are relatively simple programs to use.

HL2, EQ2, Doom 3, and practically any game performance will be quite a bit more useful, As are multi API targets.

Again why is shadermark more credible than 3dmark? 3dmark03 gave us a range of tests that at the time no other game or synthetic gave us.

Give me an FX result from a pure DX9 game with no pp hints and no shader replacement then.

credible? I'm thinking more along the lines of useful. Shader mark can test different kinds of shaders and shader rendering techniques, Each test has a specific purpose and it fills the purpose for which it was intended wonderfuly. Testing Individual shaders such as phong, ect.

3dmark03 may tell us something,. But its in a much more generic way, and IMO far less useful. Making the program pretty much useless when you compare it to programs like shadermark. Which TB has done an excellent job of keeping updated.

The FX wasnt slow in Every DirectX 9.0 game either

Why do you think that is?

Different rendering techniques, focus on register reduction, good dev relationships, And most importantly. heavy Nvidia optimisations

 

[Scali]

credible? I'm thinking more along the lines of useful. Shader mark can test different kinds of shaders and shader rendering techniques, Each test has a specific purpose and it fills the purpose for which it was intended wonderfuly. Testing Individual shaders such as phong, ect.

3dmark03 may tell us something,. But its in a much more generic way, and IMO far less useful. Making the program pretty much useless when you compare it to programs like shadermark. Which TB has done an excellent job of keeping updated.

You are looking at it one-sided. Yes, ShaderMark tests a variety of individual shaders, which is a nice way to see how the shader units perform.
What it doesn't test, however, is how these shaders would be applied in an actual game-scenario, and what the result will be, both visually and in terms of performance. This is where 3DMark is unique, and invaluable.

 

[Blastman]

... The FX wasnt slow in Every DirectX 9.0 game either,

Does the FX series actually run any games in DX9 anymore? I thought the FX series was defaulting to the DX8 path these days.

 

[Tim Murray]

The FX wasnt slow in Every DirectX 9.0 game either

well, not those with DX8 shader replacements.

 

[ChrisRay]

credible? I'm thinking more along the lines of useful. Shader mark can test different kinds of shaders and shader rendering techniques, Each test has a specific purpose and it fills the purpose for which it was intended wonderfuly. Testing Individual shaders such as phong, ect.

3dmark03 may tell us something,. But its in a much more generic way, and IMO far less useful. Making the program pretty much useless when you compare it to programs like shadermark. Which TB has done an excellent job of keeping updated.

You are looking at it one-sided. Yes, ShaderMark tests a variety of individual shaders, which is a nice way to see how the shader units perform.
What it doesn't test, however, is how these shaders would be applied in an actual game-scenario, and what the result will be, both visually and in terms of performance. This is where 3DMark is unique, and invaluable.

I dont believe 3dmark is capable of replicating a game enviroment or any synthetic test for that matter, I dont believe any synthetic test can ever achieve such a goal, So they shouldnt even strive for it. I personally dont \believe synthetics should even attempt to mimic a game enviroment, (Now keep in mind this prejudice isnt just towards 3dmark, Its always directed at codecreatures and aquamark, Even when those are game engines I find them equally useless.

The most important tests in 3dmark to me happen to be the individual ones, Whenever I did read 3dmark scores, Those were the only things I cared about, When I have used 3dmark2001SE, I compared to the 3 shader scores and fillrate, Those are the tools I have used for 3dmark in the past, I find the game scores to be ultimately useless though.

... The FX wasnt slow in Every DirectX 9.0 game either,

Does the FX series actually run any games in DX9 anymore? I thought the FX series was defaulting to the DX8 path these days.

Depends which patch you use in Far Cry ;p I believe saying the FX runs a mixture of DX 8.0 and DX 9.0 class shaders would be a better description.

 

[dan2097]

Depends which patch you use in Far Cry ;p I believe saying the FX runs a mixture of DX 8.0 and DX 9.0 class shaders would be a better description.

That actually would also describe the standard dx9 path

A heavily tilted towards dx8.0 mixture of DX 8.0 and DX 9.0 class shaders would be a better description.

I mean even on the standard path dx8 shaders still far outnumber dx9 ones.

 

[Scali]

I dont believe 3dmark is capable of replicating a game enviroment or any synthetic test for that matter, I dont believe any synthetic test can ever achieve such a goal, So they shouldnt even strive for it. I personally dont \believe synthetics should even attempt to mimic a game enviroment, (Now keep in mind this prejudice isnt just towards 3dmark, Its always directed at codecreatures and aquamark, Even when those are game engines I find them equally useless.

I really don't understand this point. What difference does it make if a game-like environment is rendered in a benchmark or in a game? Same kind of environment, same kind of workload, same kind of performance.
In fact, they aren't even synthetic tests. They are game tests. A synthetic test is a test that focuses on one aspect. That's why it's synthetic, it's not a realistic situation.
So, you are not interested in 'real' game benchmarks either then? Because I don't see the difference. And I suppose it's then your personal preference to only look at synthetic tests, and it has little to do with whether 3DMark03 or other benchmarks can replicate a game environment, if you ask me.

 

[Sxotty]

May I postulate why 3dmark is in a way bad, or any other sythetic benchmark like that.

There is a tendency of many individuals to focus on the overall score. This score however is determined by tests that are weighted according to the software developer. This is the point at which it becomes a problem. Suppose that by simply changing the weight on one test with respect to another the software developer changes who has the top spot in the benchmark. If this is the case it tends to invite if not beg for corruption and so forth because a minimal change can have a dramatic effect on businesses. Now I am not saying it is evil, I am saying that this can be a problem. A system with an invitation for corruption is a system that is not reliable.

 

[Scali]

May I postulate why 3dmark is in a way bad, or any other sythetic benchmark like that.

There is a tendency of many individuals to focus on the overall score. This score however is determined by tests that are weighted according to the software developer. This is the point at which it becomes a problem. Suppose that by simply changing the weight on one test with respect to another the software developer changes who has the top spot in the benchmark. If this is the case it tends to invite if not beg for corruption and so forth because a minimal change can have a dramatic effect on businesses. Now I am not saying it is evil, I am saying that this can be a problem. A system with an invitation for corruption is a system that is not reliable.

I think this is a "Guns don't kill people, people kill people" argument.

 

[digitalwanderer]

May I postulate why 3dmark is in a way bad, or any other sythetic benchmark like that.

There is a tendency of many individuals to focus on the overall score. This score however is determined by tests that are weighted according to the software developer. This is the point at which it becomes a problem. Suppose that by simply changing the weight on one test with respect to another the software developer changes who has the top spot in the benchmark. If this is the case it tends to invite if not beg for corruption and so forth because a minimal change can have a dramatic effect on businesses. Now I am not saying it is evil, I am saying that this can be a problem. A system with an invitation for corruption is a system that is not reliable.

Just because people use a tool wrongly doesn't make the tool wrong.

 

[Reverend]

May I postulate why 3dmark is in a way bad, or any other sythetic benchmark like that.

There is a tendency of many individuals to focus on the overall score.

Beyond3D do not cater to "many individuals". We won't/don't interfere too much with how Futuremark wants to market their products (although we offer suggestions that are more in line with our target audience). They have every right to what they think makes them more money. If they think having a "3DMarks" is important, I won't waste my time arguing with them -- I own no FM shares , but if I do, I may have a different stance. That's how it is.

Which sounds better to you -- "The Gamers Benchmark" or "The Gamers 3D Benchmark"? This relates to the what you said about "3DMarks" in a way. I suggested the latter just prior (or maybe even during) to 3DM05 going gold. You'll have to see what 3DMark05's slogan is. The slogan is probably just as important as "3DMarks" to the very folks/individuals you're talking about.

The point is "3DMarks" is essentially reflecting the overall mindset of "Bigger Means Better" without any thought given to what "bigger" really means and what it constitutes -- more sperm is better, or bigger balls is better or a bigger stick is better... it depends on both the deliverer/owner and the recipient, and what priorities each of them place on individual, er, components.

I suppose the important question is this -- if you have a higher "3DMarks" than your friend, will you have a better experience playing games you can buy now or 1 year down the road? Hard to say (specific hardware market "dominance/share" dictates how games are made in the majority of cases). But if you have higher "3DMarks", especially wrt 3DMark05, I am fairly certain you have faster 3D hardware than your friend. Unfortunately for the gamers, 3DMark03 and 3DMark05 focusses on emerging API features, instead of OOTB features (AA, AF) that gamers care a lot more about. For the 3D purists, 3DMark03 and 3DMark05 is more exciting for things beyond AA and AF.

I'm slightly tipsy, so excuse me if I have gone OT.

 

[Randell]

I'm slightly tipsy, so excuse me if I have gone OT.

I don't think you have.

I got 5500 3dmarks with my 9700pro - default bench

I get 10800 3dmarks with my 6800GT - default bench

The relative difference may be too high compared to game difference (without AA/AF) - but yeah the 6800GT can offer me twice the framerate over a 9700pro in current games.

Ignoring the V5 v Gf1 DDR/Gf2 difference which was a problem as the V5 was heavily outscored by the Gf2 in 3Dmark 2000, but nowhere near as much in games (I know why, don't tell me why) - 3dmark2001 and 3dmark03 give a good overall idea where your card sits performance wise compared to your friends, as well as giving other tools.

 

[Reverend]

One more post before I crash -- wrt "weighting" (an important matter I missed in my above post), FM actually asks their BDP members for feedback on this. All of FM's members are IHVs, save for the few media outlets like B3D. IMO, FM should have some ISV "heavyweights" BDPs that have as much interest in 3D as gameplay (eg. Carmack vs. Spector). I was asked to be the middle-man when it came to Sweeney. Unfortunately, Tim declined (for an "event" that is well known during 3DMark03's still-current life span). FM can have their ideas and "premonitions" about how games will be made in the future while taking into account latest API features (and, slightly less, latest 3D hardware features) but they're not making games for a living, and until they have input on how future games will like shape up from ISVs that become part of their BDP, it is a hard ask when it comes to relating "3DMarkXX" with "The Way It's Meant To Be Made".

It's no longer about the best hardware, unfortunately. It's about how the market is shaping up. Unless FM will come out and say forcefully "3DMarkXX is about which 3D hardware is the best at 3D rendering performance taking into account the features afforded by the latest available API at the time of each 3DMarkXX's release. It is not about which hardware will play games best in the now or in the future at the time of each 3DMarkXX's release", there will always be doubts about how much of a crystal ball 3DMarkXX is.

And don't get me started on how influential the consoles are when it comes to a dev house making games, taking advantage of 3D features and then having endless meetings anout how to develop a game whiles meeting both platforms. Game ports are expensive; dev houses would really like to spend very little cost in this. Which means thinking things over very carefully when it comes to, not only implementing 3D features, but an entire game engine. FM should probably take this into account, instead of a rather "simplistic" approach (no offense, Patric!) about 3Dmark and the PC.

Nighty-night.

 

[Moose]

I don't think 3dmark is any more or less of a valid benchmark than any other benchmark test or game currently being used for such.

I do however, think that 3dmark has been done a pretty good job of guessing things to come though.

I think the problem is that too many tend to place too much emphasis on the score you get with it on a particular piece of hardware. Maybe that is the reason so many use it, because there is a readily comparable number that you can look at as opposed to a FPS number like most game benchmarks.

 

[Sxotty]

Scali I agree to a point, but then we have gun regulation dont we .
"All FM users must be at least 18 years of age to brag about their scores "


DW as I said I was worried about how it seems a system that will invite corruption, wrt the weighting issue. Rev's post was informative, and I was not aware of how exactly they decided the weights (well I am still not aware of how exactly, but at least I have a rough idea).

The bickering about who has the bigger 3dmark score from enthusiasts is not what I was worried about. I was worried that if Dell was going to buy 300 million dollars worth of cards based on it and a tiny change could swing the contract one way or the other. Anyway I am not against FM like I said I am just worried about anything that can be so influential such as indeed D3 seems to have been portrayed to be, and I am sure HL2 performance will be. Anyway I am not on a crusade to convince anyone of anything really, just saying this might be why their is angst about FM, and as we have certain games as well.

 

[ChrisRay]

I dont believe 3dmark is capable of replicating a game enviroment or any synthetic test for that matter, I dont believe any synthetic test can ever achieve such a goal, So they shouldnt even strive for it. I personally dont \believe synthetics should even attempt to mimic a game enviroment, (Now keep in mind this prejudice isnt just towards 3dmark, Its always directed at codecreatures and aquamark, Even when those are game engines I find them equally useless.

I really don't understand this point. What difference does it make if a game-like environment is rendered in a benchmark or in a game? Same kind of environment, same kind of workload, same kind of performance.
In fact, they aren't even synthetic tests. They are game tests. A synthetic test is a test that focuses on one aspect. That's why it's synthetic, it's not a realistic situation.
So, you are not interested in 'real' game benchmarks either then? Because I don't see the difference. And I suppose it's then your personal preference to only look at synthetic tests, and it has little to do with whether 3DMark03 or other benchmarks can replicate a game environment, if you ask me.

Heh I dont see 3dmark03 game enviroments as realistic attempts at creating a game enviroment. Perhaps you missed that part of what I said, I dont see the game tests as "Game tests" at all. Of course, I'm not gonna beat the dead horse on 3dmark rendering techniques.

The only way to really determine performance is to benchmark it the hard way, Using fraps, Taking several preset runs through the game and then dividing those runs to reduce margin of error. Thats the way to simulate a game enviroment. No other way comes close, Because you're not actually trying to simulate something you cant ;p


Then again scali, you are one of the first people I've ever met who'd like to look at 3dmark's game tests as "Non Synthetic". I consider everything about 3dmark synthetic, From its game tests to its Pixel shader/fillrate tests.

 

[Althornin]

The only way to really determine performance is to benchmark it the hard way, Using fraps, Taking several preset runs through the game and then dividing those runs to reduce margin of error. Thats the way to simulate a game enviroment. No other way comes close, Because you're not actually trying to simulate something you cant ;p
.

Great.
Your method tells you nothing about how the card will perform in future games - it only tells you how it performs in THAT game.
Most people would like an indication of future performance - and in that case, arguing against 3dmark03 is gonna be tough - because it was RIGHT.

The very specualtive and cutting edge nature of its tests means that it can test things that no current game can/is. This gives you a peek, if youw ill, into the future performance of your card - in a very general sense. It let us say "in shader limited games, the R300 will be a good card, and the GFFX 5800 will not". glory be, it was right.

 

[ChrisRay]

The only way to really determine performance is to benchmark it the hard way, Using fraps, Taking several preset runs through the game and then dividing those runs to reduce margin of error. Thats the way to simulate a game enviroment. No other way comes close, Because you're not actually trying to simulate something you cant ;p
.

Great.
Your method tells you nothing about how the card will perform in future games - it only tells you how it performs in THAT game.
Most people would like an indication of future performance - and in that case, arguing against 3dmark03 is gonna be tough - because it was RIGHT.

The very specualtive and cutting edge nature of its tests means that it can test things that no current game can/is. This gives you a peek, if youw ill, into the future performance of your card - in a very general sense. It let us say "in shader limited games, the R300 will be a good card, and the GFFX 5800 will not". glory be, it was right.

If you read above, I did say that 3dmark tests do tell us something in a Generic fashion, I dont think anything could be more generic than the conclusion you gave us either ;p

My reply to him was based on his conclusion that 3dmark can simulate a game enviroment, And there are better tests for testing shader performance (As an example) Than 3dmark. In Any Event its a very circular argument, Which always turns back to my original argument. 3dmark has a limited use. But compared to other testbeds I think there are far better tools to use and focus on than 3dmark.

There was no argument that 3dmark may tell us something. However I commented earlier that what it tells us is very generic. Which limits its usefulness. I get the impression from you. That you believe we as enthusiasts are so desperate for software that limits video cards, That we'd prefer to use something like 3dmark03, Or 3dmark05 (As is upcoming) Rather than using better test beds.


So heres my conclusion on this issue:

1) Are there better synthetic tests designed to show us what current hardware can do? Or what can limit it?

I think so.

2) Are 3dmarks game tests very useful?

I dont think so at all. I never have. 3dmark05 may change my mind. But if its anything like the last two 3dmarks, Or Aquamark, then I very much doubt it. It certainly wont affect any of my buying decisions.

 

[digitalwanderer]

It certainly wont affect any of my buying decisions.

I don't think a burning bush nor a clue-by-four would change your buying decisions Chris.