Neutral Benchmarks

[jb]

3 of the 4 tests are weighted the same at 26.7%, and one test (GT4) is weighted at 20%. I don't follow you....

Errrm no I was talking about 3dmark2k3 here, from B3D own review:

3DMark03 score =
(Game Test 1 frame-rate x 7.3) +
(Game Test 2 frame-rate x 37) +
(Game Test 3 frame-rate x 47.1) +
(Game Test 4 frame-rate x 38.7)

So if I was an evil IHV and wanted to get more 3dmarks I optimize the hell out of my drivers so GT3 goes up as its the biggest bang for my efforts. If I am lucky test2/4 go up but who cares since I get more out of one test.... See this type of thinking is bad but its not a stretch to see how an IHV could follow this line of thinking...

You can lead a horse to water, but can't make him drink. Doesn't mean you shouldn't lead him to water.

What if the horse would rather have something to eat? hehehe

No one is saying you shouldn't look at and examine individual scores and individual feature tests for that matter. You can get ADDITIONAL information from them.

*edit dam web page.

Ment to say here lots of times we only see these reviews with just 3dmark scores. Thats it no breakdown. With out a score we remove some of these types of things which is probably better for Joe Sixpack.

I'm sure we'd all agree that ANY "single number" can't possibly do card comparisons complete justice. But the question is: from a practical standpoint, as a single number, is 3D Mark '03 leading anyone in the wrong direction?

My argument is why do we have to give them a single number? I could care less for how that number was calculated. My question is what does it really tell you. Not a dam thing. The goal here is to get useful information with out fluff. A final score is fluff. People will look at the final score and that's all they care about and run with it.

 

[Crusher]

]
There are one hell of a lot more DX 8 related games coming out long before primarily DX9 games hit the shelves anywhere.

Right, and I'm saying by the time those games (that use PS1.4 enough that it will really make a difference) get here, you'll need a card that supports higher than that to run the game acceptably anyway. That is, arguing that you need PS1.4 support doesn't matter, because in order to play the game, you'll need a card that's fast enough to handle it, and any such card would probably support PS2.0. Or are you saying a PS1.4 card (the only ones I know of being the Radeon 8500/LE/9000 Pro) is going to be fine on Doom 3?

]I already answered this. I just dont understand your fixation on the 8500. You are tryiing to use 1 card out of 6-8 other PS 1.4 cards to justify your arguments.

Right, except that those other 6-8 cards are either identical to the 8500 (9000 Pro), or they're not PS 1.4 cards; they're PS2.0 cards (9500, 9500 Pro, 9700, 9700 Pro), which as a rule support PS1.4, just as they do PS1.3, 1.2, and 1.1. If you're going to call the 9500/9700 a PS1.4 card, you might as well call the Radeon 8500 a PS1.1 card.

 

[Himself]

So if I was an evil IHV and wanted to get more 3dmarks I optimize the hell out of my drivers so GT3 goes up as its the biggest bang for my efforts. If I am lucky test2/4 go up but who cares since I get more out of one test.... See this type of thinking is bad but its not a stretch to see how an IHV could follow this line of thinking...

If they can really optimize and get much better performance out of those tests without cheating, I think that would be a good thing. The more code that's optimized for, the better. All having 3dmark03 around means is that all the companies have to actually test the driver code that runs the features that no games will use for over a year.

 

[Joe DeFuria]

Errrm no I was talking about 3dmark2k3 here, from B3D own review:

3DMark03 score =
(Game Test 1 frame-rate x 7.3) +
(Game Test 2 frame-rate x 37) +
(Game Test 3 frame-rate x 47.1) +
(Game Test 4 frame-rate x 38.7)

So if I was an evil IHV and wanted to get more 3dmarks I optimize the hell out of my drivers so GT3 goes up as its the biggest bang for my efforts.

No, IT DOES NOT.

You don't understand the weighting method! Did you read the ENTIRE B3D review, including this part:

The multiplication number seems a little arbitrary upon first looks, but these have been calculated by running the tests on a number of high end systems and weighting the scores such that Game Test 4 equates to 20% of the final score, while the remaining 80% is split evenly to the other three tests.

Game Test 3's FPS are LOWER than test 2 (both of which are MUCH lower than test 1) on the systems that the score was normalized on. The game test 3 score is scaled with a factor of 47.1, because on the "base system", game test 3 scored 37/47.1 = about 80% the frame rate of test 2.

In other words, if you optimize both test 2 and test 3 to each give you 5 more FPS, yes, test 3 impacts the overall score more....BUT you actually DID optimized test 3 MORE, because you are starting with a lower base FPS.

Again, the NET RESULT, is that Game 1, 2, and 3 contribute to 26.6% of the score, and test 4 contributes to 20% of the score. That's all you really need to know.

Ment to say here lots of times we only see these reviews with just 3dmark scores.

Again, not the fault of FutureMark. On the other hand, if there was only ONE score that I had to pick to use, it would be 3DMark.

With out a score we remove some of these types of things which is probably better for Joe Sixpack.

You're assuming Joe Sixpack understands or even cares to read about explanations...

My argument is why do we have to give them a single number?

Because quite simply, that's what "they" prefer.

The goal here is to get useful information with out fluff. A final score is fluff.

I see it another way. The final score is the ultimate bottom line. There is exactly no "fluff" in it at all. Because not everyone cares to see fluff, or cares to be "educated" about what all that fluff means.

There is LOTS of useful information that the "informed" person needs to have in order to make the best possible decision. There is not a single benchmark...or even a single review that I've ever seen, that has "enough" info for all consumers to be able to make the best informed choice.

 

[Joe DeFuria]

If you're going to call the 9500/9700 a PS1.4 card, you might as well call the Radeon 8500 a PS1.1 card.

We have no problems with 3DMark supporting the PS 1.1 fallback...so I don't understand the relevance of that point.

Would you prefer that there was a PS 1.1 test, a PS 1.4 test (getting zero score for GeForce3.4), and a PS 2.0 test instead? No, my guess is you'd rather there be no Ps 1.4 tests at all, correct?

 

[jb]

No, IT DOES NOT.

Run a benchmark with your hardware your drivers get X result thats your base line. Develope a cleaver way to increase GT3 only and you get a higher score vrs your base line? Yes? No?

You're assuming Joe Sixpack understands or even cares to read about explanations...

First of all if Joe Sixpack is reading the review then he is not Joe Sixpack

On the other hand, if there was only ONE score that I had to pick to use, it would be 3DMark.

Not me. I would rather use a game at least that I can understand and get a feel for it.

Because quite simply, that's what "they" prefer.

No, thats what they are use to. Just because they are use to it does not mean there is a better way. It does not mean that another way would be better. We all agree once a trend is set people tend to follow..

I see it another way. The final score is the ultimate bottom line.

Then I think we will disagree here. The score is just that a number when the more useful info left out....

 

[Katsa]

Problem with this logic is that Nvidia probably still dominates developers in terms of being the primary hardware platform. I can't quantify this, but I'm probably right.

I have some knowledge otherwise. I'd say its getting to the point of being fifty-fifty between R9700Pros and GF4Ti4600's, with programmers favoring ATI and 3d modelers favoring NVIDIA due to its stabler OpenGL drivers.

ATI I hear has done good job giving R9500Pro/9700Pro cards even to small, a bit less known developers (we got plenty too), whereas NVIDIA was a bit more reclutant with GF4's (they were really generous with GF1/2 generation cards).

 

[Hellbinder]

Right, and I'm saying by the time those games (that use PS1.4 enough that it will really make a difference) get here, you'll need a card that supports higher than that to run the game acceptably anyway. That is, arguing that you need PS1.4 support doesn't matter, because in order to play the game, you'll need a card that's fast enough to handle it, and any such card would probably support PS2.0. Or are you saying a PS1.4 card (the only ones I know of being the Radeon 8500/LE/9000 Pro) is going to be fine on Doom 3?

You dont seem to understand that all the R300/Rv350/R350 cards are just as much DX8.1 as DX9. You cant shuffle them off into the future sometime and discount them. It simply does not work like that. This includes OpenGL games like Doom-III. The R200 path, and ARB2 path both support PS 1.4 Functionality. Further ALL Dx8.1 games utilizing PS 1.4 will run faster on Cards that Support it. which includes all PS 2.0 cards. You have an expmple in 3dmark03 thats right in front of you. Any game that has DX8 shaders optomized for PS 1.4 will get a performance boost on not just the entire R200/R300/R350 based lines but the Xabre-II, Delta Chrome, PVR series 5, Nv30 Series.. and all cards into the future.

Thus it is simply NOT a moot point to have or add support for PS1.4 shaders to games comming out in the next 24 months. It makes one hell of a lot more sense that adding support for PS 1.2/1.3 which reallly offers nothing in a reason to do it.

Right, except that those other 6-8 cards are either identical to the 8500 (9000 Pro), or they're not PS 1.4 cards; they're PS2.0 cards (9500, 9500 Pro, 9700, 9700 Pro), which as a rule support PS1.4, just as they do PS1.3, 1.2, and 1.1. If you're going to call the 9500/9700 a PS1.4 card, you might as well call the Radeon 8500 a PS1.1 card.

I simply dont agree with the spin you put on this. I think my above point covers what you are saying here as well.

 

[Nick[FM]]

Not me. I would rather use a game at least that I can understand and get a feel for it.

Can you say that you _really_ understand games - how they technically work, how the engine works, what codepaths they use and why etc.. I doubt that! With 3DMark, all the info is available, if you need and want to check it up. With games, it's much harder.

Then I think we will disagree here. The score is just that a number when the more useful info left out....

Ha, but what if reviewers would post both numbers AND the 3DMark score? Wouldn't that actually be the best of both worlds? Those who don't care about the 3DMark score can read the numbers, and the ones who just want the 3DMark score, they get that. Why not both? Any special reason why not?

*edit: Assuming of course that reviewers explain the numbers and the 3DMark score. Without explanations, many readers can be left with their mouth open, and head full of questions. That applies to the review's choice of games (and other benchmarks) too. Don't think that I blame the reviewers for anything, or that I think they do a bad job! No! I just think that the choise of the benchmarks would make much more sense in reviews if there would be some small explanation included (technical and non-technical) why that benchmark was chosen.

 

[Joe DeFuria]

Run a benchmark with your hardware your drivers get X result thats your base line. Develope a cleaver way to increase GT3 only and you get a higher score vrs your base line? Yes? No?

How is this any different than developers doing "clever things" to optimize for Quake3, vs. other game tests, because Quake3 is a more defacto standard benchmark?

First of all if Joe Sixpack is reading the review then he is not Joe Sixpack

More or less agree. This doesn't mean that the score is completely meaningless though.

Not me. I would rather use a game at least that I can understand and get a feel for it.

Which game test would that be, and how does that translate into performance for games going forward? And which game documents its techniques so you know what's being stressed, etc?

Just because they are use to it does not mean there is a better way.

Agreed. That doesn't mean that they are willing to deal with a "better way" if that means they need to educate themselves to some degree on 3D Tech, so they can better understand the slew of data that's presented to them.

Then I think we will disagree here. The score is just that a number when the more useful info left out....

Yes, we may just as well agree to disagree. But I still don't think you understand the score. It's not that the score "leaves out" useful information, it's that the score is an attempt at incorporating "the most important useful info" into a single, quantitative measurement.

 

[jb]

How is this any different than developers doing "clever things" to optimize for Quake3, vs. other game tests, because Quake3 is a more defacto standard benchmark?

Its not but at least the people that play that game get results. Now ask your self this with out a score how would IHVs then handle 3dmark? Whould they try to optimize for each game test? Some of the game tests? None of the game tests? Or spend their time optimizing games? Interesting questions which I do not claim to know the answer if there was one. But keeping with a single number forces their hand a bit more. I still fault the IHV for this as they DONT HAVE to optimize... force to? Yea probably...

Yes, we may just as well agree to disagree. But I still don't think you understand the score. It's not that the score "leaves out" useful information, it's that the score is an attempt at incorporating "the most important useful info" into a single, quantitative measurement.

I think I do. Again I thing that a final score allows wiggle room for driver tweaks that may not reflect anything in the future. Any time to attempt to incorporate many results into one absolute score you are losing some info or transforming info from one source to another and there is no proof that its not a loss-less transformation I am not asking for another score or some other test to count. I want them all to be there. I want more info not less!

]Can you say that you _really_ understand games - how they technically work, how the engine works, what codepaths they use and why etc.. I doubt that! With 3DMark, all the info is available, if you need and want to check it up. With games, it's much harder.

Not to brag but I have a good understand of them as I have been coding mods for some games for awhile now. This has given me the chance to take a game/engine and strech it a bit. Push it and see where its weak points are. This has also given me access to the developers them selves as well as game engine docs that are not public which we can talk and get feedback on point of their engine/game. I Agree mod developement is not the same and the game itself but is at least on level closer....So I do have some fair understanding.

The point was if that one game was a game that I play alot then it helps me for my case

Ha, but what if reviewers would post both numbers AND the 3DMark score? Wouldn't that actually be the best of both worlds? Those who don't care about the 3DMark score can read the numbers, and the ones who just want the 3DMark score, they get that. Why not both? Any special reason why not?

Worm that would make my day. Problem is there are many sites that dont and leave it up to the reviewer to know what the hell the scores really mean. Depending on your site you maybe get someone thats good..and you may get some one just trying to get their reivew on line.

 

[nelg]

Right, and I'm saying by the time those games (that use PS1.4 enough that it will really make a difference) get here, you'll need a card that supports higher than that to run the game acceptably anyway. That is, arguing that you need PS1.4 support doesn't matter, because in order to play the game, you'll need a card that's fast enough to handle it, and any such card would probably support PS2.0. Or are you saying a PS1.4 card (the only ones I know of being the Radeon 8500/LE/9000 Pro) is going to be fine on Doom 3?

http://www.theinquirer.net/?article=7280

 

[Hellbinder]

Not to brag but I have a good understand of them as I have been coding mods for some games for awhile now. This has given me the chance to take a game/engine and strech it a bit. Push it and see where its weak points are. This has also given me access to the developers them selves as well as game engine docs that are not public which we can talk and get feedback on point of their engine/game. I Agree mod developement is not the same and the game itself but is at least on level closer....So I do have some fair understanding.

The point was if that one game was a game that I play alot then it helps me for my case

Would you mind telling us wich mods you worked on for wich engine?? Perhaps a link to the various mods websites???

 

[jb]

Would you mind telling us wich mods you worked on for wich engine?? Perhaps a link to the various mods websites???

http://www.chaoticdreams.org/

Been a member for the last few years with this team. Now I am team lead and coder (that is until I am dead or they find someone better...heheh sad Star Ship Troopers plug). I have done everything from skining, modeling, mapping and have stuck with coding. Sadly my skills in those other areas are week at best Our mod was featured on the ID Q2 Extremites CD. We then worked on UT and made it to Epics GOTYE. Our port to Q3 was not very successful (RIP ChaosArena). But I have also "dabbled" with other engines like Serrios just to see whats next. Nothing offical or published more or less just plaing with other engines to see what we could do. Our UT2k3 port of chaos is well under way. In fact I am waiting for the model team to finish up some models and then we should be ready. I know Chaos style of mod is small compared to others but we have not done too bad for just a bunch of people trying to have fun Hey at least I have gotten to ride in Tim Sweenys ferrari 8)

bah Sorry folks did not realize my number of post on this one thread..its like a record for me..eeekss

 

[Nick[FM]]

Not to brag but I have a good understand of them as I have been coding mods for some games for awhile now. This has given me the chance to take a game/engine and strech it a bit. Push it and see where its weak points are. This has also given me access to the developers them selves as well as game engine docs that are not public which we can talk and get feedback on point of their engine/game. I Agree mod developement is not the same and the game itself but is at least on level closer....So I do have some fair understanding.

The point was if that one game was a game that I play alot then it helps me for my case

Oh cool. But still, that would only be you, and a handful others. What about the Joe Sixpack's and their families? Does HE know how the game works, and what the fps in some "timedemo" means? Does HE know that 1 engine isn't performing alike in all games that it's being used? I don't think so..

Worm that would make my day. Problem is there are many sites that dont and leave it up to the reviewer to know what the hell the scores really mean. Depending on your site you maybe get someone thats good..and you may get some one just trying to get their reivew on line.

I think it would be grreat if all sites would use 3DMark03 (of course), and not only post the 3DMark score, but also post the fps and other numbers they get out of it. It would be better for all of us. In addition to that, a thorough explanation how the benchmarks works and why they were chosen etc. That would be great! Maybe some day..

 

[Joe DeFuria]

That is, arguing that you need PS1.4 support doesn't matter, because in order to play the game, you'll need a card that's fast enough to handle it, and any such card would probably support PS2.0. Or are you saying a PS1.4 card (the only ones I know of being the Radeon 8500/LE/9000 Pro) is going to be fine on Doom 3?

http://www.webdog.org/cgi-bin/finger.plm?id=1&time=20020211165445

Carmack Says:

Last month I wrote the Radeon 8500 support for Doom. The bottom line is that it will be a fine card for the game, but the details are sort of interesting.

http://www.ati.com/technology/hardware/mobilityradeon9000/testimonials.html

Carmack says:

The (MOBILITYâ„¢ RADEONâ„¢ 9000) part is the first part that we have that can actually do all of the high-end features necessary for optimal performance from DOOM IIIâ„¢. We have the vertex shaders. We have the six texture units that can do seven texture accesses, and everything that we need to do a minimal effort implementation that gets the best efficiency out of the available bandwidth. "

"You know, on our current work at Id right now, we're still pushing really hard to make DOOM IIIâ„¢ run well on various high end desk top cards. So it's pretty startling to be able to fire (DOOM IIIâ„¢) up on a laptop and see (DOOM IIIâ„¢) run at a really pretty startling good pace."

 

[Padman]

In regards to the notion that DX8 cards won't be fast enough to run the high end games coming out in the next year (Crusher?) I had already posted this in another thread, but will repeat here:

I don't believe any of the 1.4 shader capable cards will be considered "too slow" by a lot of people even 2 years from now. One of my flatmates in my student appartment has a celeron 500 with a TNT2 vidcard. He'll be taking over my 9000Pro after this summer, as I will be upgrading to something faster by that time.

He happilly plays NFS: Hot Pursuit 2 on his current system(TNT2), this is almost exactly the minimum supported spec for this game. He has to play at 640*480 with all details at the lowest and still gets what I consider crappy FPS. He however doesn't see any problem with this and even wins occasionaly from me and other flatmates with faster systems. Recently, he bought Jedi Knight 2 and is happy with that (at low detail, etc) as well.

I know several more people like him. I am quite sure he (and they) will have no problem playing Unreal 3 or whatever at 640*480 with crappy detail on a radeon9000 or similar 2 years from now.

I'm willing to bet this situation is a lot more common than hardcore gamers believe.

 

[Pete]

Two things:

One, I think Lars' second column was much fairer, and not just because he had responses from all three sides of the issue. I thank him for it, and I hope our negative feedback on his first column didn't scare him out of these fora for good.

Two, I agree that 3DM scores that are labelled as "overall" even with cards that can't render a test should be singled out in some way. It's true that 3DM tests both performance and compatability, but you can't represent both with one score. So, either 3DM should request (in its EULA) that final scores of cards that can't complete all tests are clearly distinguished by an *, or that only individual scores are presented. So jb's 7200 at a vanilla 10x7 would score a 3000*, while his 9700P at a funkdafied (4xAA, 16xAF) 16x12 would score a 4000. The * would prompt readers to investigate why the 7200's overall score is not directly comparable to the 9700P's, in which case they'd see that in individual games the 7200 was faster than the 9700P, but the 9700P would let them play game X (in full glory), while the 7200 wouldn't.

 

[legion88]

No, IT DOES NOT.

Run a benchmark with your hardware your drivers get X result thats your base line. Develope a cleaver way to increase GT3 only and you get a higher score vrs your base line? Yes? No?

Optimizing for a specific game test would naturally increase the final score. But the point is that, optimizing for GT3 has the same affect on the final score as optimising for GT2 or GT1.

Take as an example this base case:
GT1: 200 FPS
GT2: 39.46 FPS
GT3: 31 FPS
GT4: 28.3 FPS

The final 3DMark score for the base case would be 5475.

Now IHV optimizes GT3 where frame rates goes up by 10% while other games stayed the same. The final score ends up being 5621. Now do the same for GT1. It's frame rate goes up 220 FPS but the final score also ends up being 5621. Only optimzing for GT4 does the final score end up different. Optimizing GT4 by 10% yields a final score of 5585.

So it makes no difference whether you want to optimize for GT1 or GT2 or GT3.

The situation above, however, covers the ideal situation. The real world is a little different but not much.

Optimizing for GT4 generally yields the smallest benefit just like the ideal case. On the other hand, optimizing GT3 in general provides the greatest benefit, primarily because the frame rates of the other games aren't as high as the base case or frame rates of GT3 is higher than expected (don't know which, may depend on test machine).

Now the next question should be, since the GT3 optimizations provides the best benefit, is it worth it over the others?

Compared to GT1 and GT2, no. Even though there are differences, it is small. For the most bang for the buck, it isn't worth it to optimize for GT4 either.

 

[SpellSinger]

]I already answered this. I just dont understand your fixation on the 8500. You are tryiing to use 1 card out of 6-8 other PS 1.4 cards to justify your arguments. 3dmark03 shows how Ps 1.4 benefits Dx8 far more than it has any indication of DX9.

Problem with this logic is that Nvidia probably still dominates developers in terms of being the primary hardware platform. I can't quantify this, but I'm probably right.

I would say this is quickly changing. Many developers are now using Radeon 9700 Pro, especially if they are working with DX9. Nvidia does 'buy' developer seats though and will try to force their will on the ISV community. I say support ATI, especially with the quality of the Radeon 9700 and its software (great improvements over previous Radeon drivers).