Benchmarking woes, how I lost 50FPS

[marco]

Well not only reverend is finishing up his 9700pro review. I'm done also.
But somewhere along the line I got these strange results from the Hercules 9700pro. In Quake III, my max fps were about 165, even on 640*480*32 (No AA, No Ani). I thought this can't be right on my 1800XP, with 512MB ram. And nooo I did not forget to disable V-sync everywhere.

So I went looking for other reviews, Firingsquad, Wavey's review, Anandtech, Guru3d and some others. FS had this article on AMD and the 9700pro. There they listed an 1800XP performance, this should be around 230FPS. Other reviews, guru3d for example had this comments on why the performance in q3a with 9700 was so low. Well because your benchmarks result are WRONG. Guru3D.com is an example and there are others online with these bad results.

To make a long story short, it appears only in opengl and specific in q3a.
My guess is because other benchmarks don't scale that well. The problem lies in using sound with AMD based mobo's (as far as I know now, as this is what I can check here in my small test lab, I only have AMD based systems here). Onboard sound to be precise. Now we can start the discussion again about wheter you should put sound on or off when benchmarking.

Two positions on this: yes you should put it off, as you would try to mesure the performance of the 3dcard. Which is an illusion, as other components are still in your system and eating away cpu cycles (network card for example) AND you should put sound on, as no normal sane person would play a game without sound.

Status where I am now:
I have done all benchmarks without sound and q3a FPS are 220 till 1280*1024. Which is correct. When you install the onboard sound drivers: 165FPS and no way no how you get this higher. Even setting sound to low quality won't get you any FPS more. It looks like through the sound drivers the benchmark is locked on this framerate. Or that the onboard sound card and/or drivers are so bad, you loose about 50FPS. This weekend I get the Hercules Fortissimo III, then I'll check again. And I will check with an Nvidia based card.

So be very carefull when you look at reviews on different sites.

My system:
Athlon XP1800, 512MB DDR 2100 (crucial), 20GB HB Maxtor 7200rpm, 60GB HD 7200rpm both udma133, MSI KT3 Ultra mobo (realtek onboard sound chip), pioneer DVD, Philips brilliance 21" monitor, Razor boomslang mouse.

More on this, with feedback from Hercules is in my review.
Any comments are welcome.
Did I mention; what a kickass vid.card.!!

 

[Nagorak]

Sound should be disabled...no one who is buying a Radeon 9700 uses the shitty onboard sound.

But it's true, there are bottlenecks that are generally ignored in reviews. I think that's really best though, because it isolates only the graphics card which is being tested.

 

[ChrisK]

...

I seem to have a similar problem with Vulpine GLMark with my 9700.

I haven't tested this methodically, but at some point something seems to happen in my system that causes the framerates in this test (and others, like the Code Creatures benchmark that goes from like 37 fps at the 10x7 resolution to something in the range of 12-17 or so fps) to drop significantly. Last time, I reinstalled the newest drivers and it went back to normal, higher frame rates. This is on an nForce 415D mainboard.

I haven't got a clue what the reason for this might be.

Chris

 

[Randell]

Have you got the max you can out of your memory bandwidth and bios settings? i.e set at Cas 2 and bank interleave enabled etc? This made a big difference to someone elses 9700 QIIIA benches.

The problem with sound is in a lot of modern games you can have 4-5 sound options all of which have a varying impact on gaming performance. GTA3 and UT2003 neing prime examples. IMO If you use sound you should specify;

i) What sound option you've enabled
ii) What sound card you have
ii) What the performance impact is compared to no nound.

That just adds to the job and with more than one sound card vavailable as well it you might as well just disable sound to take those variables out of the equation in relation to a graphics card test, not an overall system performance test.

 

[Dave Baumann]

ii) What the performance impact is compared to no nound.

I've just tested that on an Intel motherboard review I'm working on - tested no sound, Audigy and on-board sound...

 

[LeStoffer]

I've just tested that on an Intel motherboard review I'm working on - tested no sound, Audigy and on-board sound...

... and the resultat was...?

 

[Randell]

ii) What the performance impact is compared to no nound.

I've just tested that on an Intel motherboard review I'm working on - tested no sound, Audigy and on-board sound...

excellent - and wholely appropriate in a motherboard review

 

[Dave Baumann]

... and the resultat was...?

It'll be in the review!

 

[Reverend]

My current position is that sound MUST be enabled.

For a long time, I bought into the rather silly concept that all video card reviews should not have sound enabled. That is plain wrong.

The only reason why you, as a reviewer, would disable sound is because you're comparing your scores with other scores in existing reviews and you notice "Shit, why the hell are my scores lower than the other reviews" and then you realize you enabled sound and those reviews don't even have a sound card in their machine or they disabled sound. Wrong method for a reviewer.

I don't care about the "purist" vid card enthusisasts. The purpose of a review is to inform the public about a vid card in a normal game playing scenario. If you use different vid cards in the same machine (and since we're on the sound topic, same sound card with same drivers in the same machine!), I see no reason why sound shouldn't be enabled. If, for whatever reasons, one particular vid card has a greater perf hit when sound is enabled compared to another vid card, all the more reason to enable sound - you must tell it as it is. You may even reveal "bugs" this way to the maker of the lesser performing vid card.

In a shootout masquerading as a review of one product, you can either choose to :

(a) disable sound and make the shootout even; or
(b) enable sound and STILL make the shootout even.

If your review is actually a review, and not a shootout, don't be concerned about "Gee, my scores are lower than the other reviews". It is up to the readers to question why this is so - they can question you or they can question the other review. As long as you are sure your machine is in top-notch condition (and report it as so, like a fresh OS install), why worry?

If one vid card has a much greater hit compared to another if sound is disabled, report it. I learn more this way, the makers of that card that performs lower will (hopefully) learn more this way too.

The purpose of a vid card review (even in a site that is solely concerned about 3D, which B3D really isn't) is to study performance in the most likely scenario. Not in scenarios that nobody in their right mind will play games at. Ask developers about checking compatibilities - sound enabled is always the option.

It is stupid, inconsiderate and plain old lack of common sense to disable sound in a vid card review.

I am annoyed at this "no sound when reviewing vid cards". I probably have more to say but I'll wait for the replies .

 

[Sharkfood]

I agree having sound enabled offers more value to the gamer reading reviews... but I have to disagree with it being used in benchmarks.

The reason being, I don't trust MOST website reviewers to have the mental capacity to offer accurate performance figures with just video and game options for video to consider, nonetheless what soundcard is used, what driver revision, what sound settings, what speakers (2, 4 point or 5.1?) as well as if positional sound/EAX or not.

By adding more variables into the fray that the ninnies posting fancy graphs that are ALREADY mostly completely inaccurate... it's just giving another level of complexity to them from which they have already illustrated the complexity level of existing variables is already too great for their comprehension.

If you figure an average of about 25% sway in figures based on sound settings, we are taking a delta 50% (+25% on one, -25% on another) worst case margin for error in comparisons.

I DO believe a reviewer should play games, as normal, and offer subjective analysis of a product with sound/details.. with an eye for image quality, feel, performance, bugs/issues and what not.. but fancy graphs are already popping up everywhere with completely inaccurate figures and apples->oranges comparisons, and this is without sound bemuddling something that is obviously much too complex for the average reviewer to handle anyways.

Just my $0.02

[Edit- to add a bit more to this.. you guys wonder WHY we look forward to Beyond3D reviews... It's because the chances of having 32 bit + 16 bit compared, or seeing 16x aniso scores tromping 8x anisotropy scores.. or flatline scores of 2x, 4x and 6x supersampling.. and other head scratchers gone completely unchallenged or unreasoned with.. are pretty rare.. For the love of God, don't give the Big 5 the qualification to get "Hi/Low/EAX" or "Miles Software" into the mix!]

 

[Grall]

"It is stupid, inconsiderate and plain old lack of common sense to disable sound in a vid card review. "

Can't say I agree with you there, Rev.

You're not guaranteed to use the same sound hardware in your review as the user reading it, so your "normal game playing" comparison goes straight down the tubes from the very outset.

A nforce MCP sound implementation does not load the system in the same way as a standard on-board AC97 implementation (ok, extreme comparison here), but even the Audigy and Nforce differ. You can't draw any conclusions from results obtained with one setup and apply it to another. It gets delegated to trivia-level information. "Ah, ok. Cool", sort of. But you can't DO anything with it.

Then again, most users do not have just sound in their systems. They may have firewire/usb2 cards too. Or TV tuners, or NICs or SCSI adaptors, or... And we usually run background tasks too. Distributed computing clients, chat programs, *cough* peer-to-peer file-sharing agents *cough* etc etc. Should a reviewer attempt to take all that into consideration? No of course not, that is impossible.

So why try it with sound when that is just as much a pit of snakes?


*G*

 

[Fuz]

You're not guaranteed to use the same sound hardware in your review as the user reading it, so your "normal game playing" comparison goes straight down the tubes from the very outset.

Good point.

By adding more variables into the fray that the ninnies posting fancy graphs that are ALREADY mostly completely inaccurate... it's just giving another level of complexity to them from which they have already illustrated the complexity level of existing variables is already too great for their comprehension.

Another good point.
I have to disagree with you, Rev.

 

[marco]

Well, this weekend more...
I have tried with soundoff, sound on, set sound to high and low, used 3dsound, normal sound etc, more info this weekend.
We have found some intresting info regarding UT2003 also

 

[Teasy]

I don't care about the "purist" vid card enthusisasts. The purpose of a review is to inform the public about a vid card in a normal game playing scenario.

I remember when there was a discussion over wether people should use special scripts in SS to make every card have exactly the same settings (as SS automatically changes allot of stuff per graphics card that was beyond just resolution, texture filtering and texture detail ect).

In that argument you firmly believed the scripts were important as they showed the true performance of each graphics card compared to the rest of the graphics cards in the review. My argument against the use of these scripts was that it didn't matter which card was fastest with some script. Because normal people would not use the script and a comparitive graphics card review was supposed to show which card would be fastest for the user, under normal gaming conditions and not which card had the most power in this game when every single setting (no matter how insignificant to image quality) was exactly the same.

You disagreed with that. From your comment above I take it that you've changed your mind a little bit on how reviews should be done?

I'm just curious if you would now say that using a fixed script for a game ,that no normal users would use, in a comparative graphics card review is now not correct in your opinion?

Sorry, I don't want to take this thread off topic, just interested in wether Rev's opinion has changed on this subject given the comment I quoted.

 

[Dave Baumann]

How do we define a 'normal game playing scenario' - what basis is there for saying what is 'normal'?

 

[Mephisto]

For video card reviews, sound MUST be DISABLED.

Enabling sound makes scores more system bound, something you do not want in a video card review. You wanna stress the video card and compare the video card performance, nothing else. That's why nobody benches video cards with a 1 GHz Duron.

 

[John Reynolds]

Sound should be disabled. One sound chip will average 5% CPU utlilization while another will take 10%.

 

[MikeC]

The way I've been handling sound versus nosound is in two parts. A first look at the performance of a graphics card is done with sound disabled. Then either in the same article, or a follow-up article, show performance with sound enabled. I typically opt for a follow-up article that concentrates solely on game play performance with sound enabled.

 

[Gollum]

For video card reviews, sound MUST be DISABLED.

Enabling sound makes scores more system bound, something you do not want in a video card review. You wanna stress the video card and compare the video card performance, nothing else. That's why nobody benches video cards with a 1 GHz Duron.

Even though these are valid concerns, I disagree! For video card BENCHMARKS (especially if compared to other cards like in a shootout) sound must be disabled, that much is true. For a REVIEW however it is of the utmost imprtance to give readers a lot more than just benchmarks! It should not only reflect video card performance in synthetic situations (even if they are "real" games, as long as the game is not normally played and sound is used its not a real world scenario), but also gameplay and compatibility information/impressions (both of which sound is an important part of), as well as software bundle impressions and whatever else comes to mind.

 

[Randell]

good points - obviously the answer is both!