FEAR & Quake 4 benchmarks?

And what is with this Digi, do you not see how he is being thick headed here or are you just trying to egg it on?

 

WTF Kyle, please learn to read.

I want you to reveal to us all the "fact" some benchmark profiles that come even somewhat close to your "hypothetical" where 5/6 of the frames are at one particular frame rate, and 1/6 are 3 times higher.

We all know ANYONE can come up with some pathological case for or against either of our methodologies. Please enlighten us as to why you think your pathological case has any reasonable chance of being realistically encountered. You've seen it before, perhaps? Where?

 

grmblbl not scientific calculators suck.

 

But what I stated was a fact is that your example will produce a framerate of under 30 at least once, so what you are asking me to prove is fact is something I never claimed to be fact. Learn to read, eh?

No, my "methodoligy" here is simply that the minimum framerate was reached at least once durring a test, there is no case to dissporove that.

My example was simply an extreme example of what will often happen to some extent or another when trying to extrapolate from a standard deviation. Yes the situation I presented isn't likely to turn up in practice, but show me a reasonable sample of contentious polling of framerate on various systems and I will show you exactly where the average being higher and the standard deviation being lower doesn't show that the system kept up a better framerate throughout the benchmark.

 

Let me state this clearly.

You have asserted, "using facts", that was never in question: that is, you can create a pathological case for / against either measurement methodology.

Hooray.

Now all you have to do is convince everyone that reaching a minimum framerate once is particularly interesting.

So far, so good...

Not good.

Define often. In order to assert that, you need to know the typical FPS profile of a gaming benchmark.

Um, why don't you show me that yourself, since you're convinced this is the case?

 

Standard deviation would be most useful when frame rates are normally distributed (normally in the statistical sense=gaussian distribution). Is the frame rate data distributed this way or not? That should answer the question of how useful stdev is.

I'm fairly certain that frame rate using VSYNC is not normally distributed.

My personal view is that there are probably better single number metrics than stdev, such as what percent of time the frame rate is below a key (standardized) level, which can be calculated exactly from the raw FRAPS.

 

Agreed...with vsynv enabled, it likely won't be. (Because you have an artificial "limiter" on the FPS allowed.) We're talking v-sync off benchmarks here.

Sure, I could certainly live with that.

Though the only problem with that, is standardized. Tolerable frame rates are different not only per game, but per person. So if the % below 30 FPS is given as some standard, that might be useful for Person A on Game 1, but not person B on Game 2. So, you really couldn't have a "standardized" way to do it this way, IMO. You would probably have to at least set a few different FPS levels...so you end up having more than "one number."

 

BTW, kyle, didn't take me long to find a case where your methodology doesn't work well:

http://www.hardocp.com/article.html?art=ODU1LDg=

Look at those Min FPS scores: 7, 6, 2!!

Are these stutter fests? Or gameplay issues? Apparently not, given how infrequently they occur.

 

don't you two have anything better to do?

For the record I'm with Joe, a min framemate isn't always because of memory, and you also don't need to see the min framerate to see the benefits of more video memory, some HL2 levels play faster on the 512MB x800xl vs the X850 XT and you didnt need to a see a min framerate to see the avg was higher.
That graph showing the avg min framerate I think should be a the standard.
A simply min framerate number is meaningless.

 

Joe, you questioned "how often would you say the frame rate is below 30 FPS?" I answered that with a statment of fact. Appaently you been eating paint chips or something so I see no point in trying to explain things to you any further.

 

No.

 

Yes, a meaningless one.

To make it crystal clear to you:

See Sims2 benchmarks in above link. Look at the third benchmark set. Now answer this question:

"How often is the frame rate below 5 FPS in that third benchmark of almost 5 minutes in length?"

FACT: "At least once."

Apparently, that FACT, even though a minium of 5 FPS should sound the alarm bells, is meaningless, as that bencmark represents the best quality they could get with acceptable gamplay experience.

Perhaps if you stepped back from the insults and snide comments, you could afford yourself a few minutes to understand what I'm saying.

 

Yep, Standard Deviation is indeed sensitive to absurd values, so if you have such values, or if you consider them as not mattering, you can use inter-quartile distance, or semi quartile distance (not sure about the names, I'm not native english).

With the previous case study (30,30,30,30,30,90), both give a distance (deviation?) of 0, meaning very steady framerate.
Q1=30, Q2=Median=30, Q3=30
Q1-Q3 = 0
(Q1-Q3)/2 = 0

 

I wouldn't be making insults and snide comments if you would comprehended that I never made minimum framerate out to be some holy grail and specifically stated the contrary long before you ever started contesting such an absurd claim.

 

Um, what I contend is that you are making minimum frame rate to be "holier" than standard deviation. Which I simply disagree with, hence our "debate."

Now please, if you could adress the points in my last couple posts...

 

You'd seriously rather try to convince something they're wrong?
He ain't budgin' pal...

 

I know...but it's at least passing the time watching the depths he'll go to when trying to avoid addressing relevant points...

 

The standard deviation is an indication of the variation in your group of values that you are examining. The normal distribution uses this stdev but it's not the only statistical distribution that exists & that uses the stdev. Also the normal distribution is only an estimation of the distribution, the actual distribution of the frame rate is most likely to be something entirely different.
I think it's most likely to be very asymmetric in which case it would make sense to examine stdev before and after the average. It’s very unlikely that you will find a fitting distribution for frame rate however you can always use a simple block diagram instead. You can use the stdev values for defining upper and lower variation (and you get rid of the min/max values in case they only happened during a very short time)

Code:

                                         + stdev values above avarage * cst
framerate =  average value 
                                         - stdev values below avarage * cst

/edit
if you want me to make a short example of this in excel, mail me some of those fraps results

 

Touche'

 

Again, the points in your last couple of posts were explained nearly two days ago. To recap:

http://www.beyond3d.com/forum/showpost.php?p=603900&postcount=92

That is all the holier I ever claimed it to be.

But then look at your link to those benchmarks and tell me, in those benchmarks with nearly identical average framerates, does the 7800 GT OC H2O or the X1800 XL have the higher standard deviation? The answer to that should be pretty obvious, and so should the fact that the conclusions derived from standard deviation in this comparison would contradict the actual performance demonstrated by the graph.