How benchmark 3D Cards, today

[Griffith]

I'm very bored to see false, incorrect benchmark (see the last 'opengl optimization' aka 'no real HQ' of nvidia)
I give my idea about 'good benchmarking':

FIRST, we have 3 sections:

a) Hardware Stress
b) Playable Settings
c) Eye Candy





a) Hardware Stress

every class of cards will have fitted settings, to avoid cpu bottleneck



Top Solutions (Multigpu/Xfire/SLI of top cards)

2048x1536, Maxx AA (6-8x) adaptive ON, FULL trilinear, Aniso 16x, optimizations OFF, 32 bit precision forced


Top Cards (500-600 $)

1600x1200, Maxx AA (6-8x) adaptive ON, FULL trilinear, Aniso 16x, optimizations OFF, 32 bit precision forced

Mid Cards (350-500 $)

1280x1024, Maxx AA (6-8x) adaptive ON, FULL trilinear, Aniso 16x, optimizations OFF, 32 bit precision forced

mid-low Cards (200-350 $)

1024x768, Maxx AA (6-8x) adaptive ON, FULL trilinear, Aniso 16x, optimizations OFF, 32 bit precision forced

low Cards (140-200 $)

1024x768, 4xAA adaptive OFF, FULL trilinear, Aniso 8x, optimizations OFF, 32 bit precision forced

inexpensive cards ( <140 $)

1024x768, 2x AA adaptive OFF, FULL trilinear, Aniso 2x, optimizations ON, 32 bit precision forced


in every bench we need MINIMUM and AVERAGE framerate


b) Playable Settings:

for every card, the setting where the card don't fall
under 30 FPS and the average is almost 45 fps (FPS games)
under 45 FPS and the average is almost 60 fps (racing games)


c) Eye Candy

as Hardware Stress, but with HDR enabled with MSAA (no quincux, no mixed MS-SS algoritm, pure MS or pure SS)

 

[Griffith]

another important thing is:

THE CHOICE OF GAMES

3dm 05-03-01, doom3 are old

I suggest only recent bench, with intensive shaders and fillrate requirment

1) COD 2
2) FEAR
3) 3dm06
4) Far Cry 1.4

this 4 should be in every bench, plus newer games to come, not oldest

 

[Skinner]

another important thing is:

THE CHOICE OF GAMES

3dm 05-03-01, doom3 are old

I suggest only recent bench, with intensive shaders and fillrate requirment

1) COD 2
2) FEAR
3) 3dm06
4) Far Cry 1.4

this 4 should be in every bench, plus newer games to come, not oldest

I agree, more games and driversettings as stated above.
I also like to see the min fps or better, real gameplay with fraps.

 

[BByte]

I disagree that disabling all optimizations gives the most accurate testing results.

In practically every game (especially any recent one) there are tradeoffs to be made, even with the latest and greatest hardware. For example I could prefer to have higher FPS or even turn up the AA a notch / bump up the resolution over say needlessly forcing 32bit shader precision everywhere.

Simply put, there are good optimizations and just turning them off doesn't make a benchmark more meaningful. As has been mentioned, extensive image quality testing is the only way to go really.

 

[fallguy]

I suggest only recent bench, with intensive shaders and fillrate requirment

1) COD 2
2) FEAR
3) 3dm06
4) Far Cry 1.4

Except 3dmark06 is very misinforming. Even more so than previous versions. I just went from a 3700+@2.75gig, to an Opty 165@1.8gig. And the Opty score was only 100 points less. 950Mhz slower on the CPU side, but only a tiny points difference? Please.

 

[Altcon]

Seeing as both companies optmize drivers for gameplay and benchmarks, I find [H] benchmarking phliosophy the way to go. I don't think you can really determine the workload, nor should you care, I think results count.

Subjective analysis of playabilty vs IQ should be the way to go. Benchmarking with FRAPS IN GAME and not all this synthetic BS. I really havent even bothered using 3dMark for evaluating my system since my 9700 Pro, I use it only for stability testing.

I think the way to go is to run all the cards in the same system, one configured for USE and not benchmarking, IE Antivirus and firewall in background as is customery with 99% of people using PC's (or so I hope for their sake).

Any running of Ultra High resolutions, at the settings you suggest would give no usefull information to those actually wanting to buy and use the cards for games. Maybe you can determine the technically superior card, but who really cares?

I do think extreme resolutions are good to test when Playable, if not, it's just a waste of time, as are TimeDemos. If a demo is ready to run, and it doesn't simulate all the efects and stress on the system, then it doesn't indicate any USEFULL info.

Is it good for gaming? at what settings? what are the Min FPS, Max FPS, IQ comparison IN GAME - seriously if you need photoshop to asses a problem, Then I ,for one don't care about it.

That's my take anyway...

EDIT: My post count is half diabolic now...he he

 

[digitalwanderer]

Nope, I think I'd really love to try and do a review sometime that had absolutely no benchmarks whatsoever. Purely subjective from start to finish and aimed soley at describing the experience with the card in a way that requires almost zero technical knowledge to understand yet at the same time shows the strengths/weaknesses of the part.

I think it's important to have accessible reviews too, for people who don't want/can't get too far into the technical side of things.

 

[pjbliverpool]

Seeing as both companies optmize drivers for gameplay and benchmarks, I find [H] benchmarking phliosophy the way to go. I don't think you can really determine the workload, nor should you care, I think results count.

Subjective analysis of playabilty vs IQ should be the way to go. Benchmarking with FRAPS IN GAME and not all this synthetic BS. I really havent even bothered using 3dMark for evaluating my system since my 9700 Pro, I use it only for stability testing.

I think the way to go is to run all the cards in the same system, one configured for USE and not benchmarking, IE Antivirus and firewall in background as is customery with 99% of people using PC's (or so I hope for their sake).

Any running of Ultra High resolutions, at the settings you suggest would give no usefull information to those actually wanting to buy and use the cards for games. Maybe you can determine the technically superior card, but who really cares?

I do think extreme resolutions are good to test when Playable, if not, it's just a waste of time, as are TimeDemos. If a demo is ready to run, and it doesn't simulate all the efects and stress on the system, then it doesn't indicate any USEFULL info.

Is it good for gaming? at what settings? what are the Min FPS, Max FPS, IQ comparison IN GAME - seriously if you need photoshop to asses a problem, Then I ,for one don't care about it.

That's my take anyway...

EDIT: My post count is half diabolic now...he he

I agree, I want to see cards benchmarked the way I would use them. High end cards should be tested at:

1280x1024
1600x1200
1280x700
1920x1200

All with 4x FSAA (transparency or adaptive) and 16xAF (quality) with optimisations on, as if I had a high end card, thats how I would play all games. I would only vary the resolution depending on what the game supported and performance.

I like the way Hexus benchmarks.

 

[Altcon]

Nope, I think I'd really love to try and do a review sometime that had absolutely no benchmarks whatsoever. Purely subjective from start to finish and aimed soley at describing the experience with the card in a way that requires almost zero technical knowledge to understand yet at the same time shows the strengths/weaknesses of the part.

I think it's important to have accessible reviews too, for people who don't want/can't get too far into the technical side of things.

I don't think accesible reviews are a problem, reliable ones are. You can juggle all the numbers you get off all the big sites and they're pretty easy to understand - Bigger = better.

But, the real problem (to me) is that they don't represent what you really get when you run the part in a real system, IE one that's used oin a daily basis. I've read a lot of reviews about the last few generations of graphic cards, and I've tried a couple of highend cards cards (some at home and severa l in the shop I work at). I can honestly say that the most usefull reviews I've read have been since [H] changed their benchmarking. They're also the ones who hit home when it came down to how it would feel (noise, temperature, size) and even what settings I could use it.

I love B3d's reviews, don't get me wrong, but I don't think that they're the best indication for a purchase for two main reasons:
a)They still rely on time demos for some games
b)They don't do side by side comparisons (ATi vs Nvidia)

For me as a buyer I like to see both competing products in the exact same system side by side, and not flip thorugh several pages trying to piece together the differences between systems. I know it sounds lazy, but I read a lot of reviews, and it's just more convenient.

Also since there are so many sites reviewing you have to learn where some sites favour ATi and others favour Nvidia. But as I said, I have read few reviews that actually let me know what I would really get when I brought the beast home...

Completely subjective reviews? that sounds a bit like PR. You need some empirical testing, I think FPS can acount for that, but not through timedemos or benchmarks. Real gameplay, that's the deal, it may not be apples to apples but I doubt it ever really was.
And I think that leaves room for subjective analysis of how smooth/playable etc the game was, what the IQ was like, what the difference felt like...
Well, Im sure you get it...

 

[Geo]

Purely subjective from start to finish and aimed soley at describing the experience with the card in a way that requires almost zero technical knowledge to understand yet at the same time shows the strengths/weaknesses of the part.

Oh, you want to write for [H]!

Ba dump-bump!

 

[digitalwanderer]

Oh, you want to write for [H]!

 

[Tim Murray]

It's still exactly what you described.

 

[digitalwanderer]

It's still exactly what you described.

2x

 

[FrgMstr]

http://enthusiast.hardocp.com/article.html?art=OTYxLCwsaGVudGh1c2lhc3Q=

You guys might enjoy referencing this article. While covering the 7800GS it also acted as a test of some of the issues you are discussing above.

BOO!
Another thing you might find surprising about this evaluation is that we are not going to throw a bunch of numbers at you with frame rates and charts with lots of squiggly lines. The short of it is this. You guys that have AGP motherboards do not have a lot of upgrade options in terms of video cards. So unless you are going to either get a sizable real world gaming experience boost, or an incredible value for your money, or both, there is little reason for you to even consider keeping that AGP motherboard around for today’s gaming.

As for new video editors, we have hired two in the last couple weeks, and we are looking to focus on the gameplay experience delivered by the hardware instead of tech specs.

The discussion thread that followed proved to be very interesting as well. It included a poll that contained some surprising results.

 

[Moloch]

http://enthusiast.hardocp.com/article.html?art=OTYxLCwsaGVudGh1c2lhc3Q=

You guys might enjoy referencing this article. While covering the 7800GS it also acted as a test of some of the issues you are discussing above.



As for new video editors, we have hired two in the last couple weeks, and we are looking to focus on the gameplay experience delivered by the hardware instead of tech specs.

The discussion thread that followed proved to be very interesting as well. It included a poll that contained some surprising results.

greetings mr dude who got rich off running a site for the "cool" geek without too much technical things, cuz those get in the way.
Just how many fps does it get at settings we deem playable

 

[FrgMstr]

Six years full time! w00t!!1

HardOCP now contracts 20 independent editors and folks in various other positions in three countries to help in delivering a variety of opinions. As for profits and getting rich, most of that money has been spent on legal fees and directly investing back into HardOCP over the last few years.

To address your thoughts on "too much technical things," Brent Justice might disagree with you on that when it comes to coverage. I think Brent does a great job covering the technical merits of new video card technologies. Without a doubt though, gameplay experience is our focus when it comes to video cards as we think that is what most people want to know. Please do note that we link most all Beyond3D technical write-ups so our readers are aware where they might find more in-depth reading on the topic GPUs.

 

[Altcon]

It seems that apart from making fun of "subjective reviews", no one has offered an actual opinion as to the merits and faults. Care to try?
Real easy to just dismiss it, harder to actually explain why.
Anyone?

 

[Regeneration]

Actually, these games are not good for benchmarks; you know why? Because both nVidia and ATi are targeting these games and trust me, you can do a lot of cool things with programming. Most of the games you’ve mentioned are “recommended for nVidiaâ€￾ and you can never know what kind of tricks these developers added to their games. Especially… when a lot of money is involved. The best games for benchmarks are unknown games. Every review, use another games so that way both parties will not be able to target these specific games. Good example: ATi’s magic OpenGL memory controller optimizations. I think it’s just for the reviews.

 

[RedBlackDevil]

using AA4x is just OLD

there're cards that have a bit or no hit at all passing from 4xAA 8xAF to 6xAA 16xAF

came on, we are in 2006

 

[Moloch]

Six years full time! w00t!!1

HardOCP now contracts 20 independent editors and folks in various other positions in three countries to help in delivering a variety of opinions. As for profits and getting rich, most of that money has been spent on legal fees and directly investing back into HardOCP over the last few years.

To address your thoughts on "too much technical things," Brent Justice might disagree with you on that when it comes to coverage. I think Brent does a great job covering the technical merits of new video card technologies. Without a doubt though, gameplay experience is our focus when it comes to video cards as we think that is what most people want to know. Please do note that we link most all Beyond3D technical write-ups so our readers are aware where they might find more in-depth reading on the topic GPUs.

The problem is that you focus on "playable" settings, which I disagree for a few reasons.
First- playable settings are subjective and secondly I'd much rather see how a card performs at various resolutions for obvious reasons, which is essential for a true comparison between cards.

So does your H2 fall under legal fees?

RedBlackdevil, for nvidia 4x fsaa is the max MSAA they can do in a single config, which I was hoping would be addressed with the 7XXX series.
If you go up to 8xS fsaa you get a nice performance drop so you either drop the res or make due with 4x fsaa.
However ati should really add mixed modes like nvidia for less GPU intenstive games.
I.e in live for speed I use 16X fsaa/16AF at 1600x1200 with fps that's greater than my refresh rate (75) so if i had an ati card the only thing I could do would be to enable temporal fsaa which may or may not work and if it does work it won't have the texture quality of an nvidia card.