Benchmarking woes, how I lost 50FPS

[Reverend]

Umm, I'm a gamer and I want my reviews with sound off. I don't need YOU to tell me what my choice of review is. I can do what you are doing in my own system whenever i wish to.

Hey, if you're interpreting what I have said so far as telling you what your choice of review should be, the same can be said of what Wavey (a reviewer) has said so far. We can turn things the other way around and have another gamer wanting reviews with sound on and being able to do what a reviewer did whenever he wishes to.

This topic is getting disinteresting to me (this is my last post). I will always review a video card with sound enabled. Those that think that by my doing so renders the review "irrelevant and inaccurate" insofar as information provided can go read other more "relevant and accurate" reviews, so there! "I did it myyyy waaayyyy"

Yes your highness, i know we are not worthy of your supreme video card reviewing skills, so you have to lecture us on what the right path is

Your Majesty, there's a reason for my smiley.

gkar1, the discussion so far has been just that, a discussion. I fail to see the reason for your very apparent sarcasm and dislike towards me.

 

[Reverend]

For gamers, performance with sound enable is "top dog". Which "consumer" are you?

I am a hard-core gamer that realizes that sound can effect the over-all bench and may reflect the 3d card's objective score in a bench.

Do you have some actual data to prove that performance with sound enabled is top dog?

Or is this your opinion?

You have got to be kidding me. I put "For gamers" in bold and you're telling me that a gamer playing games with sound enabled is not the normal way of playing games? I need to provide data to back this up? Jeezuz... you play games without any sound?

Man... enough, no more arguments. You guys that prefer games without sound enabled are obviously right and I am wrong.

 

[BRiT]

I'd prefer for reviews to run benchmarks with sound off first; if they have the time and space then go into the performance hit having various sound options enabled. By removing sound, the main players become CPU and Video. I believe gamers are more likely to upgrade their CPU/Video card than their sound card. It is easier for me to compare my system to a review system when there are less variables involved.

The time when I think sound-enabled is valid is when people are using a program like fraps to capture the frame-rates when playing a game. That gives a better idea how the entire system is when gaming. Using sound when running today's typical time-demos [ie: run pre-scripted events as fast as possible without involving actual cpu calculations for AI/physics] doesn't even seem indicitive of the actual gaming experience.

Once again, just my opinion, as cracked as it may be.

--|BRiT|

 

[Nagorak]

Bottomline: The performance impact for various sound cards should be reserved for Sound Card reviews. When you're testing a sound card's performance, you don't load up the game on the highest detail and run with full FSAA and Anisotropic so the game is graphics limited, what would that prove? Why test video cards any differently?

 

[SirPauly]

Rev,

Virtually all gamers play games with sound that really isn't the discussion here. The discussion is benching with sound. Playing games with sound is of course a giv'n. Benching with sound is not.

Here is my quote you pasted and was answering to:

If you're reviewing a video card, well, you are reviewing a video card and it's abilities; and performance is top dog in many consumers' minds. Why risk clouding the numbers because of something else in the system?

The view was based on benchmarks and numbers. I never raised playing games here.....numbers/benching/performance that is what I was clearly talking about.

I put "For gamers" in bold and you're telling me that a gamer playing games with sound enabled is not the normal way of playing games?

I placed "For gamers" in bold, hehe.

You obviously feel strongly about benching with sound and feel the need to go on about it. Rev....the way I did take it was like this:

In my original post I defined performance as benching.

The crux of the discussion was about benching.

You did take my quote and it offered the words performance and numbers in it.

And you defined performance as playing games without being clear considering the entire discussion was really about benching with or without sound.

I aplogize for my confusion, sorry. We defined performance differently.

 

[Geeforcer]

When you benchmark CPUs, you usually do it at low resolution with best possible video card, in order to eliminate factors irrelevant to CPU performance. No, no one plays at 640x480 or 800x600 on their P4 2.8GHz/ R9700, but if you want to find out how good a CPU really is that's how you benchmark it. Should CPU reviewers use TI4600/R9700, which few people have, or go with a 440 MX/Radeon1/GTS at 1076x768, since that's what most people have?

A game like UT2003 has what, 4-5 different sound quality options, with considerable performance delta between them. If one were to benchmark with sound, in order to accommodate how people play the game, which one should be used? When sound it introduced into equation, you have to start making assumptions about people's preferences in regard to sound quality, which are quite irrelevant to the subject at hand i.e. video card review. What sound quality setting in UT2003 do "most people" prefer - the fast but sub-par "software", the best-sounding but slow "hardware + EAX" or something in between?

To people who prefer highest quality, the benchmarks with the lowest quality are as irrelevant as ones with no sound at all, and vice-versa. And then of course we have many various sound solutions and a number of drivers for each that will provide different performance at each of the quality settings. How relevant are benchmark done on SB Live! relevant to nForce owner? What sound card do "most people" have, anyway? Do "most people" update their sound drivers on regular bases?

By the time all of the myriad variables sound introduces are taken into account, the video card review will be completely derailed. I suggest then rather then making assumption on what sound quality and what sound card people use, video cards should be benchmarked with no sound. Save sound card performance for a sound card reviews.

 

[gkar1]

This topic is getting disinteresting to me (this is my last post). I will always review a video card with sound enabled. Those that think that by my doing so renders the review "irrelevant and inaccurate" insofar as information provided can go read other more "relevant and accurate" reviews, so there! "I did it myyyy waaayyyy"

In this paragraph you've done the following:

- Show a tone of condescension towards your would be readers and thread colaborators.
- Try to ridicule and put down other reviews/reviewers as if they are not up to YOUR standards. And finally;
- Show a bit of arrogance and a thumb up our noses with "I did it myyy waaayyyy"


I was merely trying to emulate the spirit of your post with my previous remarks

 

[Reverend]

In case anyone thinks that whatever and wherever I post something is not an expression of opinion but a statement of "Reverend says this is so, so it must be so", let me start by :

In my opinion and especially in my case as the role of a reviewer :

I run benchmarks in a review as a form of indication of actual game playing performance. For a site like B3D (and as is apparent from Wavey's indication of this site's intent, focus and purpose), it is not (or no longer) the case. I don't think it is incorrect to say that neither I nor B3D are wrong - we just have different purposes.

When I write a review of a video card, I write it with one basic assumption and one sole target audience :

Basic Assumption - the person reading my review is considering a video card upgrade for playing games
Target audience - a person that plays games and is looking for a video card upgrade and is not a frequent (or even a) participant in forums (such as this or elsewhere)

That is the purpose of my writing reviews (no, it not to get free hardware ).

And then there are others that read reviews for one or two purposes not related to the above - to compare for the sake of comparing, for example. They don't have any intention to upgrade, or they do intend to upgrade but not during the time they are reading such reviews (no money yet, for example). They just read, compare this review to others and just enjoy discussing about the contents of separate reviews because they like to discuss about it in forums (since they either don't intend to upgrade or don't have the money right now to upgrade). Rightly or wrongly, I do not target such an audience in any review I write.

SirPauly, in case it is not clear to you, when I run benchmarks in a video card review, I try to replicate an actual game playing environment. That means sound is enabled. "Performance as benching" being your definition is the same as performance as game play when we use specific game demos for benching. If you don't want Quake3 timedemos, then run, say, Quaver demo, in normal mode using FRAPs to log the entire run of the demo to return an average. A normal mode run of the Quaver demo should indicate my gaming experience while playing and recording Quaver. From your post, you seem to think that benchmarks aren't indicative of gameplay performance - you're wrong, it is, as long as sound is enabled "Performance" is performance, whether in benchmarks using demos in timedemo mode or running the demo in normal mode. Am I right to say that you think "performance in benchmarks" is different than "performance in gameplay" when both (benchmark and gameplay) uses the same demo?

gkar1, it should be obvious with the way I worded the entire paragraph and coupled with my smiley that I said it in a rather joking manner... that was the first time, btw, that I have attempted to quote the way Frank Sinatra sang "My Way" as a from of light hearted attempt to get a point across. There is no condescending tone towards anyone. If I have a standard (and I do), that doesn't mean mine is "higher" than others, be they reviewers or whoever - I just have my standards and, in this thread, I have hopefully outlined them - if you don't agree with my standards, it's okay... you have a choice to choose whose/which website's standards agree with or appeal to you. Since you are of the opinion that sound should be disabled for a video card review's benchmarks, feel free to ignore reviews by me because you don't think I have provided the necessary and specific info you are looking for... I won't feel insulted . As for the apparent "arrogance" with using Sinatra's My Way, that was simply to say that every one has a choice in determining which site's reviews is appropriate in forming a decision and that my way is, well, my way and you really don't have to make any sort of decision by reading my review.

There are far, far more folks that read reviews on a website or magazine that don't know the influence that enabling/disabling sound has on performance compared to those that are knowledgeable about the "ways" of benchmarking, like, presumable, everyone that visits this forum/site.

The difference is that I take into account the high probability of those "ignorant about benchmarking ways" folks reading (or stumbling onto) my review while sites like B3D, or Anand, or Tom, or HOCP appear to assume the majority of readers of their reviews knows "THE benchmarking way" or such sites only care about their specific target audience. It is a matter of different considerations. Obviously, folks that are savvy with benchmarking procedures will already know the disclaimer of "Did this reviewer test with or without sound enabled?"... I have to assume thousands and thousands of folks jump onto the Internet for the first time everyday and may stumble onto my review... and they definitely don't know "THE benchmarking way". That is why I take great pains in explaining how I benchmark games in a video card review... again and again in every review. Anand, on the other hand, just goes straight into UT2003 benchmark graphs without any explanation of details.... the assumption is there that the benchmarking procedure is already known.

I don't target the guys that regularly visit this forum... I have to think of those that want the best video card for playing games AND assume that these folks know jack shit about "THE benchmarking way" if and when they happen to read/stumble upon my reviews. If B3D or Anand or Tom are to assume they are a "specialist" website and go be-damned-with-these-folks-that-don't-know-about-benchmarking-video-cards, all power to them and I don't see anything wrong with that honestly.

 

[Dave Baumann]

All these talk about different CPU utilizations by various sound cards is nice and good info - but actual framerates is more important.

Well, I would have thought the point would have been farily obvious – the larger the CPU utilization of the sound board the more time the CPU is going to be spent dealing with sound, not dealing with the game rendering elements, hence the larger the FPS hit. When games are CPU limited on current boards without sound, a sound board that uses 20-30% of CPU cycles is going to have quite a negative impact on performance (as Mikes benchmarks illustrate).

Hopefully the many gamers that currently visit this site will not gradually be disinterested or they become like you, changing from a gamer to a 3D enthusiast. Again, I'm not saying this is a bad thing. You seem to have described the direction B3D is taking. Hopefully John won't feel bored

Why would they? Those elements will still be there – they can just skip the benchmarks they don’t like if needs be. Plus, I didn’t say that B3D would only do tech reviews. Reference boards / first looks should be tech reviews, there’s scope of other directions in board variant reviews – which is why I don’t mind John taking a different tack with his 9700 review and Marco with his.

Hey, if you're interpreting what I have said so far as telling you what your choice of review should be, the same can be said of what Wavey (a reviewer) has said so far.

No, theres a vast difference there Rev – I’m saying “This is how my reviews will be, take it or leave it†the way you are saying it is “Hey, you’re a gamer so you must agree with my way of doing thingsâ€, without even asking them.

If B3D or Anand or Tom are to assume they are a "specialist" website and go be-damned-with-these-folks-that-don't-know-about-benchmarking-video-cards, all power to them and I don't see anything wrong with that honestly.

I don’t think any of them are specialist websites, just they just take an scientific approach to reviewing the particular product they are reviewing by eliminating as many of the variables as possible.

 

[Evildeus]

I don’t think any of them are specialist websites, just they just take an scientific approach to reviewing the particular product they are reviewing by eliminating as many of the variables as possible.

Well, all reviews have some specifics, and without taking the exact same specific you can't obtain the same results. Taking this card or this hard drive etc. have some important consequences on the result (want to compare a xp 2800+ with a nforce 2 and nforce 1?).

I play with a specific motherboard, a specific HD, a specific RAM memory, etc. So i think that saying that one approach is "more" scientific than another is not scientific .

Personnally i play with sound on, and i would prefer a review with sound enable. So when i look at a bench i prefer something that is +/- 5% around the score i should get than 10% (sound disable) above plus the variance. But it doesn't bother me miore than that because i know it .

http://www.hardware.fr/articles/415/page8.html
http://www.hardware.fr/articles/415/page8.html

Just my 2 cents....

 

[Dave Baumann]

I play with a specific motherboard, a specific HD, a specific RAM memory, etc. So i think that saying that one approach is "more" scientific than another is not scientific

Well, its about removing as many variables as possible - we can't review a graphics card without a CPU, motherboard and RAM. These are required variables, we can do it without a sound card.

 

[Reverend]

No, theres a vast difference there Rev – I’m saying “This is how my reviews will be, take it or leave itâ€￾ the way you are saying it is “Hey, you’re a gamer so you must agree with my way of doing thingsâ€￾, without even asking them.

Again, your (mis)interpretation. All I have said thus far is the same "This is how my reviews will be, take it or leave it" (the "I did it myyyy waaayyy"). There is, of course, a difference in assumptions about what you and I think folks "should want" or should be the correct way (with or without sound). You gave your explanations about why sound be removed and I gave my explanations of why sound shouldn't be removed.

[edit]In summary, every reviewer will do what they think is correct. I think I am, you think you are, hopefully everyone will be confused as hell

 

[Evildeus]

Well, its about removing as many variables as possible - we can't review a graphics card without a CPU, motherboard and RAM. These are required variables, we can do it without a sound card.

Of course you can do, but do you think that saying config 1 without sound disable can be compared with same config is different that saying config 1 with sound enable can be compared to the same config?

Then what's the point arguing that 1 is more something that another one. For the rest, well each reviewer can do as they want, and surely the pure VC is good and if some reviewers add benchs with sound it gives a little plus

 

[jb]

I like to see sound on the reviews. Why? this way I can look at my system specs and my benchmark results and know if my system is close to other systems with the same specs when I run my games the way I play them (ie with sound on). I guess as long as the reviewer states if they had sound on/off then its all good. Theory is all nice but real world test I feel is more valuble to me. However there is no one right way as its different strokes for different folks (Ok Simon I left my self open...care to take a shot). Again IMHO...

 

[WaltC]

My current position is that sound MUST be enabled.

For a long time, I bought into the rather silly concept that all video card reviews should not have sound enabled. That is plain wrong.


It is stupid, inconsiderate and plain old lack of common sense to disable sound in a vid card review.

I am annoyed at this "no sound when reviewing vid cards". I probably have more to say but I'll wait for the replies .

I agree completely. What does running a benchmark without sound tell you in the first place? Other than providing you--in some cases--with frame rate numbers 98% of the gaming public will never see (because turning off the sound never occurs to them and they'd rebel against the suggestion they "should" play that way), not much else. Now, if the game had no sound, then it would be appropriate, seems to me.

I kind of think it is interesting that in the UT2K3 read.me Epic states that if one doesn't use the latest Audigy drivers but runs with an earlier set (aside from those just released--in August, I think) he might see a huge fps hit--I forget but I think Epic says 40-50% or some similar gigantic number--if EAX 3.0 and 3D sound are enabled. However, I noticed that when running UT2K3 with the latest Audigy drivers my fps dip 1-2 fps, and that's just a guesstimate--it might not actually dip at all.

OK, here's the Epic quote so you don't have to look it up:

If using a 3D sound accelerator such as the Sound Blaster Audigy series
sound card, you can go into "Options/Audio" to enable EAX 3.0 by selecting
the "Hardware 3D Sound + EAX" options.

You need to upgrade to the latest version of Sound Blaster Audigy drivers
in order to get acceptable 3D sound performance.

Using Unreal Tournament 2003 in conjunction with earlier versions of the
drivers MAY cause severe performance problems (major slowdowns in the
order of 30-50% while playing sound) in which case you should change the
option back to the default "Software 3D Audio".

If your computer is hooked up to a 5.1 speaker system, you should go into
"Options" and turn on "Hardware 3D Audio" to take advantage of 360-degree
sound panning, which rocks.

I'd say that's pretty strong incentive to get the new drivers.

Sure, UT2K3 is morbidly cpu-stunted (a deliberate attempt to build in engine longevity?), but it seems to me that often, with the right equipment and the right drivers and the right software, the actual differences in running with and without sound are minimal at most. In my system at home I certainly can't see a difference between software and hardware 3D sound in UT2K3. (It hasn't occured to me to turn sound off, though.)

So maybe I could accept 3D card game benchmarks which benchmark both with and without sound, both with and without 3D hardware sound. (More work, but still if precision and information are the goals...) But a straight benchmark with no sound...and that's it? Ack--I would dismiss it out of hand, because I don't play that way. I can't imagine being unique here.

If even a straight videocard comparison uses *game benchmarks* without sound--only, with no other numbers--I'd say the numbers were bogus (not inaccurate--just irrelevant.) It's fine with synthetics, of course, because they don't use sound to begin with with. Just my four cents.

 

[PSarge]

I agree completely. What does running a benchmark without sound tell you in the first place? [...snip...]
I kind of think it is interesting that in the UT2K3 read.me Epic states that if one doesn't use the latest Audigy drivers but runs with an earlier set (aside from those just released--in August, I think) he might see a huge fps hit--I forget but I think Epic says 40-50% or some similar gigantic number

...so if Wavey, or Rev, or whoever conducts a 9700 review with those broken sound drivers installed, what have they just tested? The graphics? No, the system was limited by the sound. The graphics card was sitting there, twiddling it's thumbs, waiting for the sound card to get out of the way.

Doesn't strike me as a fair test of the graphics card.

(It's an extreme case, but it illustrates the problem. Sound should be off)

 

[Joe DeFuria]

Basic Assumption - the person reading my review is considering a video card upgrade for playing games
Target audience - a person that plays games and is looking for a video card upgrade and is not a frequent (or even a) participant in forums (such as this or elsewhere)

OK, I've been silent long enough.

I can appreciate that basic assumption and target as the purpose for your reviews, rev.

However, I don't think your current format meets the goal of serving that target.

What if:

1) The person reading your review is not only considering upgrading to the card you are testing, but a competitive card?

and / or

2) The person reading your review does not play the games that you are testing.

You seem to frown upon "head-to-head" comparisons. But I don't think it's very often that a person says "should I upgrade my card X to card Y?" It's ofen, if not usual, that he says "should I upgrage my card X, and if I do, should I get Card Y or Card Z?"

So IMO, if your target audience is in fact gamers looking to possibly to decide whether or not to upgrade, your method of not doing "head-to-head" is a bit flawed to begin with.

That being said, and more on topic...

I can also appreciate enabling sound in a review so that you can relay a "more realistic" absolute performance expectation of that product. Absolute performance is important for evaluating, well, absolute performance. However, because you cannot know what games your readers are interested in, and because your readers are likely considering OTHER products as well, absolute performance on a set of games with one video card is not the sum of information needed to make an informed choice.

Relative performance is also needed. If there's a trend that one card "relativley" out-performs another card, not matter what the absolute numbers are, that is important to know. And turning sound off (and otherwise minimizing non video-card bottlenecks by running games in video card high resolutions, etc.) is the only "fair" way to relatively compare graphics performance.

In sum: there is a place for BOTH "absolute" and "relative" benchmarking. And IMO, both of these are needed to satisfy the needs of your target audience.

 

[SirPauly]

SirPauly, in case it is not clear to you, when I run benchmarks in a video card review, I try to replicate an actual game playing environment. That means sound is enabled. "Performance as benching" being your definition is the same as performance as game play when we use specific game demos for benching. If you don't want Quake3 timedemos, then run, say, Quaver demo, in normal mode using FRAPs to log the entire run of the demo to return an average. A normal mode run of the Quaver demo should indicate my gaming experience while playing and recording Quaver. From your post, you seem to think that benchmarks aren't indicative of gameplay performance - you're wrong, it is, as long as sound is enabled "Performance" is performance, whether in benchmarks using demos in timedemo mode or running the demo in normal mode. Am I right to say that you think "performance in benchmarks" is different than "performance in gameplay" when both (benchmark and gameplay) uses the same demo?

I understand that your reviews you try to represent a gaming experience for your reader and why the hell I have respected you for many years Rev. I understand why you want to bench using sound and is a very valid view but my point was it still is a subjective view -- that's all. I think benching with no sound just respresents what a 3d card truly can do without anything that may skew the number results -- that's all. Not to demean you in anyway. Discussion is about points and counter points and things may get confused a bit because of clarity in our words at times.

On the discussion on how we define performance it depends on context. I was strictly talking about benches when it comes to performance numbers but benchmark numbers doesn't translate into game-play and if this is your point -- I agree with you 100 percent. Game-play performance is more than just numbers in a bench and any gamer knows that, imho, and why you're trying to simulate game-play in your specific benches you create.

 

[demalion]

Hmm...well, there are some basic principles behind the scientific method. You eliminate what you can (and obviously leave what you cannot) as variables for analysis. This allows the behavior observed to be more compartively meaningful. I.e., if you are trying to test a coffee machine you'd do some testing on plain water instead of introducing the variable of trying to control the concentration and type of coffee you'd test with...

OTOH, this is not the same thing as observations that are useful in a practical way. If you were going to test a coffee machine you just bought for yourself, you'd likely test it with coffee, because as far as YOU are concerned, you have a limited subset of coffee and coffee concentrations you are likely to be interested in.

One is scientific, the other is practical, and both are useful.

Which is more useful for a review? For comparison, I'd say the first is, simply because that is the purpose of minimizing variation: to reduce the factors in that comparison (in the case of a video card review, to try and make the video card as significant and as isolated a factor as possible).

For determination of what a user will see on their own system, the latter is...assuming the variables introduced are applicable to that user.

The ideal (to me) is to use the first for all video card centric comparisons (resolution, video features), and then at the end compare with a common factor to make the results more practical...i.e., show comparitive benchmarks at a selected resolution for each benchmark (since each game will possibly behave significantly differently with sound) with and without sound with a likely common sound card (i.e., SB Live! to be most useful at this point I think). To catch something like video card A suffers more of a performance hit than video card B with a certain sound card would take a more indepth analysis than either approach in isolation, but is it really worth it to do this much extra work when the variation the user will actually see depends on so much else? For instance, there is no guarantee a SB Live! will offer the same performance hit on different chipsets for the same CPU...

Actually, I don't have a big interest in seeing the sound comparison step at this site for a videocard review...I don't run the settings reviewers use, have the same type of system, so I don't really care about matching the fps I see in reviews. I'm only interested in a thorough, hopefully meaningful, comparison to determine the strengths of cards comparitively in regards to video performance, isolated as much as possible. Sound comparison in that context is just, forgive the pun, introducing noise into the data.

BTW, some people that have hacked Audigy drivers to work with the SB Live! have reported decreased CPU usage...my current understanding is that the "efficiency" of the Audigy in this regard is more than coincidentally due to Creative's driver policies with regards to their cards (i.e., pay them money to get better drivers).

 

[Fuz]

Bottomline: The performance impact for various sound cards should be reserved for Sound Card reviews. When you're testing a sound card's performance, you don't load up the game on the highest detail and run with full FSAA and Anisotropic so the game is graphics limited, what would that prove? Why test video cards any differently?

Exactly!

I think we need to differentiate the words "review" and "benchmark".
When reviewing a vid card, leave sound on, play heaps of games (not just one or two for half an hour. God knows how people can call that reveiw!), spend some 'quality time' with the card and give us your subjective report. When benchmarking a vid card, turn sound off. In fact turn anything off that would hold the vid card back from performing at its optimal.

Fuz