Defining framerates and methods of describing framerate fluctuations *spawn

[HTupolev]

Ok I understand. But if your that focussed on the variations you probably want to avoid dividing and deriving numbers, instead you just publish the tables of values (e.g. 1fps to 30/60fps) and show how many how many seconds in the capture period were at 30fps, 29fps, 28fps and so on.

I'm not sure you do; the issue here isn't how we record the distribution of "fps" values (i.e. as a value in the video, in a table, etc), it's how we measure said "fps" values in the first place. The "dividing and deriving", as you call it, isn't some kind of extraneous filtering that's being imposed upon already-measured "fps" values; it's how "fps" is measured based on when frame boundaries occurred. You don't have the "fps" values without, as you call it, "dividing and deriving numbers."

//===============

Let's suppose that over the last second, we had a new frame at 100ms, at 200ms, at 300ms, at 400ms, at 700ms, and at 900ms.

One way to calculate fps is to say that 6 new frames occured in the last 1 second, 6/1 = 6fps. This gives us the general rate over the second, but it doesn't clearly spell out that we recently had a big drop.

Let's use a "sharper" filter instead, and measure framerate over half-second intervals. If we decide to take a measurement at 500ms into the second, we note that 4 new frames showed up in that .5 seconds, 4/.5 = 8fps. We can also take a measurement at the end of the second; we note that 2 new frames showed up in that latter .5 seconds, 2/.5 = 4fps. Now, using this "sharper" filter, we can see that the framerate was higher in the first half of the second than the second, and now have a clearer view of the performance drop.

This is what I'm talking about.

 

[Shifty Geezer]

instead you just publish the tables of values (e.g. 1fps to 30/60fps) and show how many how many seconds in the capture period were at 30fps, 29fps, 28fps and so on.

Taken at what frequency?!

How are you calculating your fps numbers that you are putting in that chart? Are you going to count how many frames shown in one second, and get a new reading every second? Are you going to count how many frames per second but count them at quarter-second intervals, creating 4 fps values per second? Are you going to count how many frames per quarter second and take a new reading every quarter second? Or how much time since the last frame and turn that into a fps value?

The only untainted data is pure frame-interval, a table of thousands of entries. That's clearly not a nice format for human readability, so we compress that data into an average, but that average takes a slant based on how it's derived. To DF's credit, as well as the rolling fps average, they also show every single frame time in a graph at the bottom of their vids. It's pretty detailed. The only shortcoming is that a representative figure, frames per second, isn't particularly accurate, but that's the single data point people latch onto (because it's the one in nice human-readable form!).

 

[DJ12]

You could use a percentage instead.

Table could be as follows:

Total frames in test: 50000

Time at 30: 98.1%
Time at 29: 1.8%
Time at 10: 0.1%

Surely this will give people a better indication of how bad a framerate problem actually is. This way people can work out exactly how many times the framerate drops to each level and it would also give an indication of how bad the problem really is.

Of course it'll do nothing for DFs incorrect conclusions, but it's a start.

 

[Shifty Geezer]

Time at 30: 98.1%
Time at 29: 1.8%
Time at 10: 0.1%

Aaaaaaaaah!
How are you calculating your 30 fps and 29 fps and 10 fps??

 

[London Geezer]

Sorry but I lost you all a long time ago. What is going on here??

 

[pMax]

...discussing where seconds start and end, it seems.
Yeah, statistic is a fascinating disciple

 

[Rodéric]

Duration is measured in time unit, the international unit for time is the second.
What you want is to have the duration of every frame recorded and build graphs/tables from that data.

 

[London Geezer]

Would I be naive in believing that what we already consider Frames Per Second is enough to know whether a game is smooth or not?

 

[iroboto]

Sorry but I lost you all a long time ago. What is going on here??

They are debating the validity of digital foundries testing method.

Scientifically speaking we should have more than one source to measure against DFs claims.

And there are claims in this thread that DF is wrong with their conclusions. That they are cherry picking clips to create false conclusions for ???? (Unknown Reason)

However those claims in this thread have no empirical data that can be achieved somewhat repeatably for us to actually dispute DF with.

And now you are here.

 

[Deleted member 11852]

Taken at what frequency?!

The frequency of sampling should always be able to capture all variances of the data being measured, so 60 samples per second on a console. This should be captured for the duration of whatever the test (capture) period is. You should never be compromising on capturing the raw data. If the sample data isn't accurate then resulting your numbers will not be truly representative.

One way to calculate fps is to say that 6 new frames occured in the last 1 second, 6/1 = 6fps. This gives us the general rate over the second, but it doesn't clearly spell out that we recently had a big drop.

And this is my biggest issue with the discussion. People keep talking about calculating the framerate but the framerate doesn't need to be calculated it can be accurately recorded and expressed with no loss of the subtle variations of the actual (not calculated) framerate during play over the capture period.

I'll leave this here because I think we're fundamentally at different places but it's interesting to note that, I think, DF are not recording and reporting frame rates as I believed they were.

 

[Billy Idol]

Naively, the only reasonable thing to do (for me) is the following:

t=0 is the start of the clip you want to analyse
t = n with n being a natural number are the time intervals we measure, meaning every single second we update our statistics...no more, no less (there is imo really no reason to increase or decrease this interval for the actual fps beyond 1 second):

- for every single second n we count the number X of unique frames during this second.
- after the second has passed, we simple use X as our actual frames per second at time n (although it is the value after the second has passed, so somehow a delay on screen) and display this on the screen.
- in addition, we add up all those X and divide them through the actual value of t, which is n, this gives an average value of the fps X_avg = sum(X)/n.
when we reach the end of the clip, this number gives us the average fps of the total clip length, also displayed on the screen.
- furthermore, we also determine the minimum and maximum value of X, X_max=MAX(X_max, X), which we also update every single second and display this on the screen. again, after the clip is finished, we get the global minimum and maximum fps over the length of the clip.

The best case scenario would be X_max = X_min = X_avg, which can only be reached when X is constant for all n.

The only parameter left would be the total length of the clip. This of course has an impact on the average fps you get for the total clip. The longer the clip, the better the average value X_avg in the sense of statistics. However, as we know, this average value might not be a good measurement for perceptual smoothness during playing. The length of the clip wouldn't have an influence on the minimum and maximum and no influence on the actual fps X distribution on the other hand.

Maybe another parameter would be, which game situation do you measure: just staring in the sky without moving, driving through the GTA city, heavy fire fights, walking through thousands NPC, etc etc.

But I think, this would make DF fps comparisons much better and more valuable for comparisons of different games. I especially think that PC games would come out quite bad with such an analysis, as in my experience on my PC, the average X_avg and the extremes X_min and X_max are quite quite different!

 

[Shifty Geezer]

Sorry but I lost you all a long time ago. What is going on here??

How does one define the fps framerate of a game? Some people are talking about frames per second without appreciating that they are an average sampled across the data, meaning you have to choose a sample length and interval. Those parameters change the resulting average, so the same data can present different results. Reporting and making sense of those results requires more info than just an fps number.

Would I be naive in believing that what we already consider Frames Per Second is enough to know whether a game is smooth or not?

I'd say so. 59 fps without vsync will be smooth but with tearing. 59 with vsync will be 60 fps most of the time but with doubled frames every now and then causing a very noticeable judder. Describing that judder in terms of framerate, it represents a drop to 30 or 20 fps for one frame. Thus you could have a framerate of 59 fps with a range of 0 (every frame takes 1/59th of a second to render) or a range of 40 fps, with highs of 60 fps and very short-lived lows of 20 fps. Frames per second as an average doesn't describe smoothness.

The frequency of sampling should always be able to capture all variances of the data being measured, so 60 samples per second on a console.

On a half second gameplay clip, the framerate == 0 fps?

This should be captured for the duration of whatever the test (capture) period is. You should never be compromising on capturing the raw data. If the sample data isn't accurate then resulting your numbers will not be truly representative.

What's the correct rolling interval? Should a new average be created for every discrete second of video, so you might have 60 fps for one second, 45 fps for another, 52 for another? Or should you overlap intervals so you create an average for time intervals 0..1s, 0.25..1.25 s, 0.5..1.5s, 0.75..1.75s, 1..2s, etc? Or should you just average the entire gameplay clip and use a different sample length for each measurement?

Hypothetical case, what if there's a game that renders 30 unique frames in half a second and then zero updates the second half? The average across the one second interval would be 30 fps, but the actual experience would be nothing like that. A rolling average showing a variance from 60 fps to 0 fps would be far more descriptive and accurate.

And this is my biggest issue with the discussion. People keep talking about calculating the framerate but the framerate doesn't need to be calculated

It's calculated. If you count a number of frames and divide by time interval, you have a calculation, even if its a division by one. Or, from another POV, you never calculate it but measure it, only you don't need to measure across a second. Only people aren't going to want 'frames per half second' and similar, so we scale them to the standard time interval.

 

[Shortbread]

I think when DF makes statements that "system A" gameplay is smoother / frame-rates are better than "system B"... because they found a cherry picked scene where "system B" dropped 2-3 frames for a few seconds, it must mean "system A" is running the game in question better. When in reality, "system B" handles frame updates/transitioning better during testing.

Example: I noticed a certain game that DF was testing that had a target frame-rate of 30fps, yet the visual graph showed one system with a consistent 30fps, while the other showed a slightly less stable 30fps from a visual standpoint. (looking at the graph). Not enough of a drop to register it at 29fps (on the fraps like counter), but noticeable enough to see on the graph it wasn't a consistent 30fps. In other words; a perfect straight line versus a slightly squiggly line.

As I stated before, I think it's DF wording at times that gets console warriors panties in a bunch... not so much the testing. Personally, I think console gamers should only worry when frame-rates are really in the shi**er, rather than these rare drops or slightly torn frames. Pointless noise, IMHO.

 

[Goodtwin]

I'd say so. 59 fps without vsync will be smooth but with tearing. 59 with vsync will be 60 fps most of the time but with doubled frames every now and then causing a very noticeable judder. Describing that judder in terms of framerate, it represents a drop to 30 or 20 fps for one frame. Thus you could have a framerate of 59 fps with a range of 0 (every frame takes 1/59th of a second to render) or a range of 40 fps, with highs of 60 fps and very short-lived lows of 20 fps. Frames per second as an average doesn't describe smoothness.

You must be ultra sensitive if one doubled frame per second causes you to perceive a "very noticeable judder." I personally notice the judder effect when Vsync games dip down in the 40's and 30's, but seeing as how I have played MK8 extensively, I can attest that the judder effect in a 59fps game is very very minimal. The judder effect does become noticeable as the framerate drops, but to say 59fps is not smooth does not reflect the experience I had with MK8.

 

[Shifty Geezer]

Every time a frame is doubled, it creates a massive interruption to the flow. It's not judder, but a clear break in the smoothness. The worst cause of judder is irregular framerates like a 3:2 pulldown of 24 Hz movies for 60 Hz displays and it's harder for the brain to 'sync' to the irregular timing. That sucks. 24 Hz updates are much smoother and don't judder. But throw in a repeat frame and it's pretty obvious. If they are common, it'll just be an annoyance. As per the DF analysis of MK8, they immediately knew something was amiss in the silky-smooth 60 fps. I accept different folk will have different levels of perception and responsiveness to various display qualities and a doubled frame at 60 fps might not grate, especially if it's regularly paced, but a doubled frame can't ignored.

And most importantly, we were originally talking about a duplicated frame at 30 fps dropping framerate for that one-frame interval to equivalent for a 20 fps refresh, which is a much lower 'framerate' even though the proportional change in interval is lower (an increase of 50% from 30 fps to 20 fps, instead of a 100% increase from 60 fps down to 30 fps for a doubled frame).

 

[London Geezer]

I was literally just posting that if I understand it correctly it's like the old (and not so old) 3:2 pulldown but you beat me to it. That was very annoying but you're saying that one single frame would affect gameplay that much?
I hate framerate drops like the next guy but I'm unsure how many doubled/dropped frames I'm seeing which affect me.

 

[function]

Every time a frame is doubled, it creates a massive interruption to the flow. It's not judder, but a clear break in the smoothness. The worst cause of judder is irregular framerates like a 3:2 pulldown of 24 Hz movies for 60 Hz displays and it's harder for the brain to 'sync' to the irregular timing. That sucks. 24 Hz updates are much smoother and don't judder. But throw in a repeat frame and it's pretty obvious. If they are common, it'll just be an annoyance. As per the DF analysis of MK8, they immediately knew something was amiss in the silky-smooth 60 fps. I accept different folk will have different levels of perception and responsiveness to various display qualities and a doubled frame at 60 fps might not grate, especially if it's regularly paced, but a doubled frame can't ignored.

Absolutely! 3:2 is a great example of how even a steady and perfect seeming 24 fps is anything but, and you have to look a the actual frame timing and the changing rate of cycling through the frames to actually see what's happening.

3:2 pulldown is awful, and all the more awful because it's a regular patter. Once you see it, you look for it, and you can't make yourself 'unsee' it. Some people never see it, and they are the lucky ones. So yeah, even a 'perfect' frame rate can hide important things.

.....

This reminds me of something related to 'frame pacing', actually. On the Dreamcast, while most games handled the drop from 60 Hz NTSC to 50 Hz PAL by changing the internal timing or simply running slow, a couple of 60 fps games handled the drop to 50 fps by simply skipping every 6th frame. This meant that the game was running slow for five frames, effectively at 83% speed (5 frames in 100 ms instead of 6), then for the next frame (which should have been the 7th) the game effectively sped up to 167% and 'caught up'. So despite a 'smooth' vsynced 50 fps, and the game averaging the correct speed, it felt awful to look at and I could not unsee the constant tick tick tick tick when things were moving.

Until I swapped to VGA (forcing 60 hz output from the game) I simply didn't want to play it Virtua Fighter 3 TB.

Long story short: Everybody Hz, sometimes.

 

[Deleted member 11852]

On a half second gameplay clip, the framerate == 0 fps?

No, the answer is still what I wrote above: The frequency of sampling should always be able to capture all variances of the data being measured, so 60 samples per second on a console.

If you're only sampling for half a second the frequency of samples should remain constant, it's only the duration (and therefore the number) of samples that will be halved. Just because your measure for less time does not mean you should compromise the frequency that data is captured.

The goal of DF articles, as regards to framerate analysis, is to give the reader an idea how the framerate (measured in frames per second) is over the course of testing, so the notion of sampling for short periods like a second or half a second, would introduce a known, but unquantifiable bias if used to "calculate" frame rates. Again, this is my sticking point here. Report frame rates based on empirical data, it should not be calculated from small samplings as a second or half a second.

That is why DF test for many minutes or hours although they occasionally do focus on short segments where they thing it's interesting to do so, e.g. engines under stress.

What's the correct rolling interval? Should a new average be created for every discrete second of video, so you might have 60 fps for one second, 45 fps for another, 52 for another? Or should you overlap intervals so you create an average for time intervals 0..1s, 0.25..1.25 s, 0.5..1.5s, 0.75..1.75s, 1..2s, etc? Or should you just average the entire gameplay clip and use a different sample length for each measurement?

This is a running mean but this approach is typically used when you wish to normalize short-term fluctuations in the data with the goal of highlighting longer term data trends. If the intention is to capture accurate framerate changes, particularly over short periods (you keep mentioning half a second), then using this is contrary to your goal.

Personally I would much rather be told the game ran at 30fps for 17% of the time, 29fps for 60% of the time and so on. Averages are awful, particularly when there is a frame rate limited (say 30fps) on the game. Medians are only slightly better. Having produced a lot of Government statistics, I'm sensitive to how unrepresentative they can actually be.

Hypothetical case, what if there's a game that renders 30 unique frames in half a second and then zero updates the second half? The average across the one second interval would be 30 fps, but the actual experience would be nothing like that. A rolling average showing a variance from 60 fps to 0 fps would be far more descriptive and accurate.

Frame rate and frame output stutter are different things, although of equal importance. Which are we discussing?

 

[Deleted member 11852]

Everybody Hz, sometimes.

 

[HTupolev]

Personally I would much rather be told the game ran at 30fps for 17% of the time, 29fps for 60% of the time and so on.

Since you seem to find the notion of determining framerate by taking (number of new frames in last deltaT)/(deltaT) offensive:

Given a particular point in time, how are you proposing that we determine what the framerate is?

That is, if I have a 60fps video recording of a game coming off a console (so I've captured all the video data that the console is outputting), how do I determine what the framerate of the game is at, say, exactly 3 seconds after the start of the video?