Benchmarking woes, how I lost 50FPS

[WaltC]

...so if Wavey, or Rev, or whoever conducts a 9700 review with those broken sound drivers installed, what have they just tested? The graphics? No, the system was limited by the sound. The graphics card was sitting there, twiddling it's thumbs, waiting for the sound card to get out of the way.

Doesn't strike me as a fair test of the graphics card.

(It's an extreme case, but it illustrates the problem. Sound should be off)

Turning the question around, what sort of graphics card test would it be if they tested with broken videocard drivers?

I mean, come on, people doing reviews have to know what they are doing, and I'd say there's no better chance of them doing it with broken sound card drivers than with broken videocard drivers. I mean, if I can read the read.me in the UT2K3 help folder, don't you think they can, too?...

Besides, what I was saying was that it would be interesting to see:

1) Without sound
2) With sound (EAX & 3D)
3) With sound (software 3D)

..etc...And of course this would only apply to *game benchmarks* anyway, which deal with games 98% of the gaming populace plays with a sound card turned on.

In some cases, as I've said, it would give you a chance to see whether sound impacts a game's frame rate performance or it doesn't, and if it does, then to what extent? I see almost no fps degradation in UT2K3 using EAX 3.0 & 3D--but that's just me looking at the fps numbers on the side of the screen as I play. I haven't gotten into any more depth about it.

You can argue the issue either way--if running with a sound card runs the danger of shifting the focus of the review to sound (which a good reviewer doesn't have to allow), then running without sound surely presents an unrealistic picture of a given videocard's performance since most people run with sound on.

 

[Geeforcer]

Turning the question around, what sort of graphics card test would it be if they tested with broken videocard drivers?

A good one, actually, it the author makes a note of the problem. The quality of video card drivers is very much relevant to the subject at hand (video card review). The quality of the sound card drivers is not.

 

[Reverend]

I think we need to differentiate the words "review" and "benchmark".
When reviewing a vid card, leave sound on, play heaps of games (not just one or two for half an hour. God knows how people can call that reveiw!), spend some 'quality time' with the card and give us your subjective report. When benchmarking a vid card, turn sound off. In fact turn anything off that would hold the vid card back from performing at its optimal.

Wow, a video card review without any benchmarks! What will NVIDIA or ATI do to show their hardware is better than the other?

 

[LeStoffer]

Going kinda off topic here: Where are the bloody reviews Marco, Rev and ben6!?

 

[Gilhooley]

I really miss review benchmarks with both vsync and sound. The difference between highs and lows can be quite shocking.

Afterall, it wouldn't hurt to see some real scores. This "early adopter" business get old real fast. Like mouse movement and vsync on my new 9700, yay

 

[Schreck]

2) The person reading your review does not play the games that you are testing.

This is a critical point that you raise here. When benchmarking video cards, you have to realize that you're testing maybe 4 game benchmarks but the user might be playing hundreds of different games with the video card they purchase based on your review. For this reason, it is critical to give them the information they're looking for: which video card is better?

Game-specific sound idiosyncracies, particularly for games you may never actually play, don't factor into that decision.

 

[marco]

LeStoffer: You're right, where are the reviews. Well with a little luck it will be up this weekend. That is my target. I will conduct some extra benchies with this dicussion in mind. I'm going to pick up my Hercules Fortissimo III today.

 

[Crusher]

The question of whether or not to enable sound has nothing to do with how you normally play the game. It is simply determined by what you are trying to show through your results. If you want to compare two or more video cards, it doesn't matter if sound is enabled or not, as long as the same setting is used on all cards. If you want to show how fast a single card is capable of rendering the graphical aspects of a program, you need to disable sound and any AI/physics you have access to. If you want to give an accurate portrayal of how a particular card will perform while playing a specific game, sound should be set to whatever values it would normally be used while playing the game (typically ON, with the best available hardware 3D options enabled). In this situation, it might be prudent to include multiple benchmark sets, each using a separate sound card, much like people are used to doing with processors.

 

[demalion]

The question of whether or not to enable sound has nothing to do with how you normally play the game. It is simply determined by what you are trying to show through your results. If you want to compare two or more video cards, it doesn't matter if sound is enabled or not, as long as the same setting is used on all cards.

I disagree with this item...with, for example, the SB Live! and its well known bus usage, the impact on performance with different graphics cards on different motherboard chipsets may not be ignorable. For this comparison it is better to use no sound to remove doubt from the equation. It is for this reason a comparison of hit for a particular popular sound card would be "ideal", but to me more properly should be part of a sound card comparison review.

The impact of the sound card is practical info, but "scientifically" picking one sound card/motherboard interaction with video cards/benchmark programs to represent all possible sound card/motherboard interactions is an invalid precept at root, self-defeating for the intent of including sound in the first place. Either limiting it to a point of reference for each benchmark at the end and representing it accurately as not representing all sound cards/motherboards, or conducting a thorough set of comparisons (which to me belongs in a sound card comparison review, possibly extended with testing with various motherboard chipsets over time, perhaps as part as the motherboard review process) is the only way, IMO, to make it a comparison not prone to blindly ignoring unforseen complications with these interactions.

Either it is a review focused on videocards solely as much as possible, or it attempts to provide more practical and useful information beyond that and must take the extra steps to accurately achieve this end. Again, in my view, focusing on video cards solely is enough of a challenge for the comparison part of a review (some reviews don't have comparisons, just benchmarks, or are limited to more directly comparable comparisons like cards in the same family/using the same drivers, and so benchmarks with sound seem to not raise the same issues IMO).

 

[Pete]

My two cents

As long as the cards are compared on the same system (and what kind of a review would it be if they weren't?), I'd prefer they be benched with sound, as it's more realistic. The 220/165 framerate difference is purely academic, because you're not going to play a FPS of all games without sound. I don't buy video game cards for academic reasons, and I don't read video card reviews based on whose has the biggest numbers. I look for correctly conducted and interpreted info, and enabling sound doesn't invalidate that. A theoretical advantage of one video card over another in ideal conditions may become moot when system limits are taken into account (slow CPU/RAM, inefficient sound drivers), and that info is more important to me than blind benchmarking.

Of course, reviewers may not have the time to test the various sound + video card combos for performance discrepancies, so I'll understand if they choose to remove the sound variable, as I'm not paying for the info they provide. But I don't think it's a realistic assesment, as I can't expect to achieve those framerates in my machine, as I won't play games without sound.

 

[Dave Baumann]

But I don't think it's a realistic assesment, as I can't expect to achieve those framerates in my machine, as I won't play games without sound.

So, Pete, what is your system at the moment?

 

[Pete]

Hmm, I should've said, "I can't expect to achieve similar framerates in a similar machine, if I'll be playing with sound and the reviewer benches without." And part of why I read video card reviews is to get an idea of real game framerates, so I'd prefer that sound be enabled. I'm guessing "real game framerates" is the (chief) philosophy behind the CPU-scaling benches B3D and AnandTech provide, and I wholeheartedly approve.

My system is way behind the curve, an aging P3-866 with an Xpert 128/Voodoo 2 combo and a cherry on top (Aureal SQ2500--remember when people actually benchmarked soundcards?). Obviously I don't stray much higher than CounterStrike in system requirements, and I've even stopped playing that b/c <20fps in firefights just isn't fun. I don't buy many new PC games, though, so I'm sure an 8500LE will be enough to satisfy me as a cheap upgrade. I spent maybe half of my PC budget on a nice monitor (Diamond Pro 900u, $700 at the time ), so my priorities weren't speed at all costs back when I built this PC. Heck, I'd probably still be running (hobbling, really) a Cel266@400 if I hadn't of picked up this P3-700@866 cheap due to an Egghead.com blunder.

I still keep up with the latest news/rumors for my next machine. Still, I've got enough of a backlog of games to play that I'm not sure I'll be building a new PC for gaming anytime soon. Maybe I should stay out of these 220+fps conversations altogether--I barely hit 70fps staring at a wall at 6x4 in CS.

 

[Randell]

well tbh a new machine will make even older games look so much better and fly - Tribes 2 for example is a revelation to me now compare to when it come out (Duron950/V5 v XP1800/8500)

 

[Nagorak]

well tbh a new machine will make even older games look so much better and fly - Tribes 2 for example is a revelation to me now compare to when it come out (Duron950/V5 v XP1800/8500)

Still runs like a dog even on top of the line systems though...

 

[Randell]

well tbh a new machine will make even older games look so much better and fly - Tribes 2 for example is a revelation to me now compare to when it come out (Duron950/V5 v XP1800/8500)

Still runs like a dog even on top of the line systems though...

I get 50-70+ fps on my system.

 

[Dave Baumann]

Hmm, I should've said, "I can't expect to achieve similar framerates in a similar machine, if I'll be playing with sound and the reviewer benches without." And part of why I read video card reviews is to get an idea of real game framerates, so I'd prefer that sound be enabled.

But this is my point Pete - you've started to add caveats after I asked that question. In reality reviewers systems are so far removed from most of the audience that they don't really bear much 'real world' relavence to them in the first place.

 

[Pete]

In reality reviewers systems are so far removed from most of the audience that they don't really bear much 'real world' relavence to them in the first place.

Really? I suppose if the reviewer is OC'ing his system via the FSB multiplier, their results wouldn't be too "real world." But I'd think most reviews use stock, if high-performance, systems. If you think most reviews aren't being realistic in testing cards on systems way beyond what their audience have, then perhaps they should use more realistic systems (or include CPU scaling benches).

OTOH, considering most people won't buy a new high-end video card at launch, perhaps it's better to review with bleeding-edge systems, as those same systems will become mainstream alongside the tested bleeding-edge video cards, thus keeping the review relevant down the line.

If your point was simply I shouldn't expect reviews to offer realistic framerates, only relative ones, that's not something I'd like to accept (because, in that case, there are too many caveats on the review ).