FEAR & Quake 4 benchmarks?

No, they weren't, and still aren't.

Right...you're saying that the single minimum frame rate tells you, uh, the single minimum frame rate? Got it. And that's all you wanted to say?

How about getting to the point I'm trying to ask...why is that particularly useful if you don't know how often it's reached? If one benchmark has a "low" being reached 1 time in 5 minutes, vs. another one not reaching it at all over 5 minutes...

By eye-balling it? Couldn't tell. Looks pretty close.

?? How you figure that? The GT has a higher peak, but the XL is above 30 FPS in the latter stages of the benchmark much more consistently.

Um, by my estimate I would say that the actual performance shown by all of the graphs is pretty close. (Which, BTW, is the point of HardOCPs methodology..)

 

Yeah, that is what I said in that bit I just quoted and have been saying for the past few days. Blowing my comments up to mean more than that and asking me to defend such a stramwan is pointless.

It isn't particularly useful, it is only as useful as I claimed it was.

By eyeballing it, but obviously we have different eyballs. I'll shoot Brent an email to see if he might be able to provide the full results so we can check our estamates.

Sure, fairly close but with the x1800xl coming out a bit roughter. Hopefully we will soon be able to see how that compares to the results of standard deviation.

 

I am again left at wondering what your "point" is.

My point is that when combined with average frame rate, standard deviation is a better metric than a single minium frame rate (combined with average frame rate) when trying to judge "playability."

 

Well hopefully Brent will respond to my email with the information I requested, at which point I will try once more to eplain my possition.

 

Why do you need information from Brent to explain your position? Is your position:

1) Minimum frame rate very useful.
2) Minimum frame rate is more useful than standard deviation
3) Standard deviation is more or less useless

Some combination of the above or something completely different?

 

Without the average and minimum, the SD is pretty useless IMHO. But that is not to say that the SD isn't useless when combined with average and minimum.

You could have a five minute run through from Battlefield 2 with the frame rate cap left intact averaging at 60 frames per second, around 18,000 frames are rendered (if my maths is correct). Say there's a standard deviation of 10 frames per second on the run through, you will still have around 90 frames that are likely to be below 30 frames per second, as 99.0% of frames are within 3 standard deviations of the mean (30-90fps). Those 90 frames that are 'below' 30fps are likely to be in heavy firefights where you actually want a frame rate above 30fps.

I think (& hope) my maths is right.

That's where the minimum becomes important - those 90 frames could dip well below 30fps, and that'd piss me off when I'm gaming. At least when I'm reviewing - I aim for less than 0.1% of the frames rendered to be below 30fps in order to consider something 'smooth and playable'. In five minutes of game play @ 60fps, that'd equate to less than 20 frames dropping below the 30fps threshold.

 

heh, I did a small test, using fraps (easy program btw)
running 3DMark03 game test 1

http://img445.imageshack.us/my.php?image=asymmeticfpsdistribution1qd.jpg

I use the method as I described in my previous post, SD is very important imho, it's the only way to figure out in what FPS ranges the gros of the game test was played

 

you need to use

 

Sounds a bit like a quartile "deviation".
How did you select the "treshold" value (to choose what goes in your right and left parts) ?

 

Something different that I have tried to explain to you mutiplie times but since it obviously isn't sinking in I am going to hold out until I have some actual benchmark numbers to illustrate my point.

 

http://www.badongo.com/file.php?fil...ribution__2005-10-25_ass+verd+fie+probeer.xls

need to sleep, if you want to take a look at it

 

I disagree on that point. I'll try to explain why:

Take that graph you choose yourself from HardOCP again. We have two short dips and one longlarge peak. http://www.hardocp.com/article.html?art=ODU1LDg
Suppose we flip that graph vertically (around the 25fps line) so we have two short peaks and one large dip.

I think we all agree that such a long and large dip would make the game unplayable. (or at least would seriously hurt playability). while the very short dips in the original graph would not (or only minimally) hurt playability. Agreed?

But when I flip the graph, the std dev stays exactly the same value!!

So how will you ever determine if a game is playable from that value?
That std dev value is the same for both situations, and with just the average value and the std dev value there is no way to determine which of the two graphs apply.

The situation is even worse. You could have graphs were the amplitude of the peaks and lows is less, but the duration is longer. Depending on whether I flip the graph the playability would be even better or even worse than the above situations. And again, the
average value and std dev would be exactly the same values.

So what does minimum value tell us? Well, if you ask me, the minimum value should be seen as a boolean value. You either have a value below 25fps or not.
If lower, then you get stutters, but you have no way of knowing how bad or how much.
If it is higher than 25, then you don't have stutters. It's not much information, but it's better than nothing.

Thinking about this graph I have changed my opinion on the usefullness of average combined with minimum and std dev. It gives you so little information that it's not worth the effort.


A better solution seems very simple and I think someone already suggested something like it before.
Instead of two extra values for std dev and minimum, reviewers should show three extra values.
Those values would be: percentage of time below 25fps, percentage of time between 25fps and 60fps, percentage of time higher then 60fps.

So in this graph it would be something like 3%, 85%, 12%.
When I flip the graph like I did before it would be 12%, 85%, 3%. So it already gives me MUCH more information than both std dev and minimum.
When the amplitudes are lower and the duration longer you get e.g. 5%, 75%, 20%.

The only thing that it doesn't show me if how high the peaks are and how low the dips.
But I don't care about that. Whether a dip is 10fps or 5fps is not really interesting. Both give stutters and that is what counts in real life.

Why choose 25 fps and 60fps?
Some people think anything lower than 60 fps is "too low". Others think that 50fps is still fine, and only values below 25fps are too low.
Almost everybody falls in either of those categories. I have never met someone who uses 50fps or something like that as borderline.

This methods gives us a lot of information. It's easy to explain to readers, and it is still quite possible to add it to a bar graph with averages.

So what do you all think? Good solution? Shall we just forget about standard deviation and minimum values?

 

Anyone want to post some actual results with the competing methods. Be eaier to see it in practise rather than hear the theory.

 

3D Performance with Quake 4: Part 2 Mainstream Cards

RV530 doesnt get such a big boost, we knew that already didnt we?

 

fillrate limitation?

 

thats what I assume

 

Ok one last time for those who didn't read earlier:

Standard Deviation suffer from aberrant values, if you want to discard those, use Quartile Deviation instead, or even Semi Quartile Deviation.
Standard Deviation (applied to FPS) is all about frame rate steadiness, which is something desirable (variation in framerate makes the game feel "jerky" (not smooth))

Quartile Deviation (assuming that's the correct english name):
Compute Q1, Q2 and Q3, Q2 is the Median, Q1 is the Median of the first subset (left of Q2), Q3 is the right subset's Median (right of Q3).
Quartile Deviation = Q3-Q1
Semi Quartile Deviation = (Q3-Q1)/2

In the case study earlier:
30, 30, 30, 30, 30, 90
Average = 40
SD = 22.36 (rounded) (Interval [17.64, 62.36])
Median = Q2 = 30
QD = 0 (Interval [30,30])
SQD = 0 (Interval [30,30])
(Q1=30, Q3=30)


Now does anyone have a REAL case with a low sample count so we can check what's best

 

but your average is now lower.

Because you combine the average with the standard deviation to get an "idea" of what frame rates you are fluctuating between.

When you "flip the graph", you will have a lower average...combine that with the same standard deviation, and you will make the assumption that that you will be dipping to lower frame rates.

 

Try again explaining concisely.