Frame Rate Analysis Thread (Simple Rules Post #2)

[grandmaster]

Unstable framerate would create screen tearing easily if there's no V-lock.

It's obvious that if GTA IV wouldn't have Vertical Synchronization Lock, the game would be full of screen tearing.

Then it must have v-lock as I've now double-checked frame-by-frame three of the tests (which were around 900 frames each) and there is not a single torn frame amongst them. Also worth noting that all the Eurogamer vids are running a 60fps capture at 30fps (ie 50% speed) and I don't see any screen-tear there either, although I concede that I haven't checked those frame by frame. No frames are being omitted though, it's running at half-speed and still v-lock is not an issue.

The EGTV versions of the Eurogamer vids are showing 544 lines out of 720, so a strong representative sample.

PS3 Race Driver GRID is an excellent case to work on though, as the almost-Ubisoft levels of v-lock tear are a textbook example.

 

[Jedi2016]

Interesting idea for a thread. I look forward to seeing it develop into the sister-thread for the Native Resolution thread.

 

[grandmaster]

OK, I've done some work behind the scenes with Quaz and fpsdetect has been refined, so I'm very confident we can soon post results. We've got the detector working on a 1280 pixel one-line check, and worked on the tolerance level. We can now adjust the detection level. You'd think that for uncompressed video, we shouldn't need to, but more on that in a later post...

So we're very close to posting results.

Looking at the preliminary results, in the case of Race Driver GRID, the game appears to be running at 1fps to 4fps slower on PS3 depending on the clip.

But here's the thing: in-game PS3 is not v-locked... 360 is. However, on the pre-event intros, neither is v-locked, yet the PS3 is obviously pumping out more torn frames.

Is there any value in measuring the amount of torn frames in any given clip?

 

[goonergaz]

Then it must have v-lock as I've now double-checked frame-by-frame three of the tests (which were around 900 frames each) and there is not a single torn frame amongst them. Also worth noting that all the Eurogamer vids are running a 60fps capture at 30fps (ie 50% speed) and I don't see any screen-tear there either, although I concede that I haven't checked those frame by frame. No frames are being omitted though, it's running at half-speed and still v-lock is not an issue.

The EGTV versions of the Eurogamer vids are showing 544 lines out of 720, so a strong representative sample.

PS3 Race Driver GRID is an excellent case to work on though, as the almost-Ubisoft levels of v-lock tear are a textbook example.

so clearly this is never going to be 100% because GTA does have tearing in places and without measuring those parts 100% "like for like" (which let's face it is almost impossible anyway due to no of cars/time of day etc) then these results will always be subject to inaccuracies. You only have to see that one of your GTA tests put PS3 above X360 f/r to prove the point really...no 2 tests are ever the same unless it's something like the 3D tests for gfx cards (scripted)...I remember Doom 3 PC having things you could run to get your FPS score...that's the only way to get an accurate score.

 

[grandmaster]

I agree that in a best case scenario we should be comparing like with like 100%. Where we can't have 100%, the next best thing we can do is produce multiple samples and/or extend the length of the sample so we get more of an average. With regards GTA, the intro at least will be entirely like for like.

The objective with the GTA results was to back-up the evidence of my own eyes that the PS3 game doesn't feel as solid to play. Each test is hardly empirical, but as close as conditions would allow. Combined they show a trend and tries to add a bit of science to what is otherwise a case of 'this is what I think...'

The tests will be re-run with the new detection technique but first indications are that the headline figures have barely shifted at all.

 

[bagofsuck]

So is gears 2 really 30fps? because the gameplay was so facts sometimes..

 

[Shifty Geezer]

Gameplay and framerate aren't linked together. Gears2 looked sub 60 fps - 60 fps is buttery smooth and nothing I've seen for this title was like that. Framerate looked higher than 30 fps at times as there was screen tearing on occasions, but the motion blur along with the editing added to the sense of fast-and-frantic. I expect framerate to match Gears 1.

 

[betan]

grandmaster, have you tested performance implications of Xenos upscaling to 1080p?
(Possible with a game that doesn't match at 720p, 30fps and v-locked.)
It would be interesting to have a confirmation that it's indeed free, not even 0.5% overhead which would still be hardly noticeable.

Gameplay and framerate aren't linked together. Gears2 looked sub 60 fps - 60 fps is buttery smooth and nothing I've seen for this title was like that. Framerate looked higher than 30 fps at times as there was screen tearing on occasions, but the motion blur along with the editing added to the sense of fast-and-frantic. I expect framerate to match Gears 1.

Did you get that from tearing on 30 fps encoded video?

 

[grandmaster]

grandmaster, have you tested performance implications of Xenos upscaling to 1080p?
(Possible with a game that doesn't match at 720p, 30fps and v-locked.) It would be interesting to have a confirmation that it's indeed free, not even 0.5% overhead which would still be hardly noticeable.

This is a very interesting idea. I can capture 1080p at 60fps in YPrPb 4:2:2, which still provides plenty of precision for frame comparison. 24-bit RGB would takes that sample down to 30fps (although I'm planning a driver revision to give full precision at 60fps).

So what we are looking for is a game that's a solid 720p30 and maybe has a replay feature I can use to take multiple captures of the same video.

Any ideas? A saved replay from PGR4 perhaps? Any other games so we can get a larger array of samples?

 

[Quaz51]

This is a very interesting idea. I can capture 1080p at 60fps in YPrPb 4:2:2, which still provides plenty of precision for frame comparison. 24-bit RGB would takes that sample down to 30fps (although I'm planning a driver revision to give full precision at 60fps).

So what we are looking for is a game that's a solid 720p30 and maybe has a replay feature I can use to take multiple captures of the same video.

Any ideas? A saved replay from PGR4 perhaps? Any other games so we can get a larger array of samples?

yes you will can test upscaling impact on X360 but also on some PS3 game, compare 720p and 1080p framerate on some PS3 game can to be interesting, on GTA for exemple

 

[betan]

This is a very interesting idea. I can capture 1080p at 60fps in YPrPb 4:2:2, which still provides plenty of precision for frame comparison. 24-bit RGB would takes that sample down to 30fps (although I'm planning a driver revision to give full precision at 60fps).

So what we are looking for is a game that's a solid 720p30 and maybe has a replay feature I can use to take multiple captures of the same video.

Any ideas? A saved replay from PGR4 perhaps? Any other games so we can get a larger array of samples?

Dirt?
Halo 3? it is already upscaled, can be used to compare 720p upscaling to 1080p.
I think a game with longish cutscenes like GTA4 may be useful too, not as much as a long race replay though.

And for PS3's software upscaling Burnout Paradise and GTA4 may be interesting,
BP is first software upscaling, then hardware.

 

[Shifty Geezer]

Did you get that from tearing on 30 fps encoded video?

It probably came out a bit wrong. bagofsuck suggested Gears 2 being 'fast' (sic facts) was pointer to >30 fps. I was saying the vids don't show this because 60fps is wonderfully smooth. And yes, the footage released has been 30 fps, so it couldn't show it even if it was! The tearing was evidence of my expectations that they wouldn't aim for 60 fps. 30 fps gives you twice the amount of time for rendering. 60 fps is very constrictive. Thus if you want a lot happening without big frame fluctuations, you'll choose the lower framerate. Or disable V-sync and have tearing. Gears 1 was locked to 30 fps, with no vertical sync when framerate dropped below that. that makes more sense, but what I was seeing was smoother than the standard 30 fps. However, upon rewatching I don't see the tearing, so I think was something whack with QuickTime.

I still expect framerate to match Gears 1; 30 fps max to avoid tearing, no v-sync when framerate drops to keep framerate as high as possible. I don't think Epic will, or can, get the improvements across the board they're after as well as halving the time they have to do it!

 

[grandmaster]

Yes, I'll get onto the 1080p test later. In the meantime, thanks to some great behind-the-scenes help from Quaz51, here are the first frame rate results.

Race Driver GRID (demo)

Like-for-like comparisons
These tests are taken from the pre-event cut-scenes, generated by the game engine. Aside from the camera wobbling differently, everything is the same.

Test1- BMW Event Intro
360: 30fps (6.72% torn frames)
PS3: 28.226fps (39.52% torn frames)

Test2- Nissan Silvia Event Intro
360: 30fps (no torn frames)
PS3: 30fps (no torn frames)

Test3- Mustang Event Intro
360: 30fps (no torn frames)
PS3: 29.748fps (11.76% torn frames)

In-game tests
Basically the event is allowed to start, car accelerates full throttle until just before it hits scenery. Scenery is like-for-like, car AI introduces a variable here. But with two tests full of cars and AI in effect, it's a decent enough indication of performance cross-platform. Note that in Test5, there are no CPU cars as it's a drift challenge.

Test4- BMW Race Start, accelerate forwards
360: 30fps (v-locked)
PS3: 28.76fps (32.38% torn frames)

Test5- Nissan Silvia Race Start, accelerate forwards
360: 30fps (v-locked)
PS3: 30fps (18.07% torn frames)

Test6- Mustang Race, accelerate forwards
360: 30fps (v-locked)
PS3: 26.722fps (39.79% torn frames)


Methodology then. The fps detector counts the number of duplicated frames in its sample. The actual amount of screen being measured can be preset. To measure a precise fps, only one line of 1280 pixels from the centre of the screen is sampled.

To measure the amount of torn frames, the whole screen is sampled. Dupe frames from this test minus the dupe frames from the one line test are worked out as a percentage of the whole sample. This should work on any 2VBL game, like Race Driver GRID appears to be. Bear in mind that this percentage is of the entire 60fps output of the console, not of the measured frame rate.

In the case of GRID it also appears to be the case that the whole game isn't v-locked on PS3, and the more challenging intro sections on the 360 version can also tear. However, in-game 360 appears to be v-locked.

So how sensitive is the scanner? Will sampling just one line fool the scanner if the actual screen is completely static to the human eye? No, the scanner is actually so sensitive that even when the camera is static and there's no perceived movement at all, we can still detect a 30fps update. Here we are in GRID, at 0mph.



Sampling just the top line of this video, here's the difference between each frame in the first second of the video:

frm 0: diff from frm 1 = 0.0000% at (0,0)
frm 1: diff from frm 2 = 0.0037% at (0,0)
frm 2: diff from frm 3 = 0.0000% at (0,0)
frm 3: diff from frm 4 = 0.0249% at (256,0)
frm 4: diff from frm 5 = 0.0000% at (0,0)
frm 5: diff from frm 6 = 0.0166% at (128,0)
frm 6: diff from frm 7 = 0.0000% at (0,0)
frm 7: diff from frm 8 = 0.0058% at (0,0)
frm 8: diff from frm 9 = 0.0000% at (0,0)
frm 9: diff from frm 10 = 0.0116% at (0,0)
frm 10: diff from frm 11 = 0.0000% at (0,0)
frm 11: diff from frm 12 = 0.0145% at (128,0)
frm 12: diff from frm 13 = 0.0000% at (0,0)
frm 13: diff from frm 14 = 0.0266% at (256,0)
frm 14: diff from frm 15 = 0.0000% at (0,0)
frm 15: diff from frm 16 = 0.0012% at (0,0)
frm 16: diff from frm 17 = 0.0000% at (0,0)
frm 17: diff from frm 18 = 0.0087% at (128,0)
frm 18: diff from frm 19 = 0.0000% at (0,0)
frm 19: diff from frm 20 = 0.0262% at (256,0)
frm 20: diff from frm 21 = 0.0000% at (0,0)
frm 21: diff from frm 22 = 0.0162% at (128,0)
frm 22: diff from frm 23 = 0.0000% at (0,0)
frm 23: diff from frm 24 = 0.0224% at (256,0)
frm 24: diff from frm 25 = 0.0000% at (0,0)
frm 25: diff from frm 26 = 0.0204% at (256,0)
frm 26: diff from frm 27 = 0.0000% at (0,0)
frm 27: diff from frm 28 = 0.0141% at (0,0)
frm 28: diff from frm 29 = 0.0000% at (0,0)
frm 29: diff from frm 30 = 0.0253% at (256,0)
frm 30: diff from frm 31 = 0.0000% at (0,0)
frm 31: diff from frm 32 = 0.0253% at (256,0)
frm 32: diff from frm 33 = 0.0000% at (0,0)
frm 33: diff from frm 34 = 0.0150% at (128,0)
frm 34: diff from frm 35 = 0.0000% at (0,0)
frm 35: diff from frm 36 = 0.0096% at (128,0)
frm 36: diff from frm 37 = 0.0000% at (0,0)
frm 37: diff from frm 38 = 0.0158% at (128,0)
frm 38: diff from frm 39 = 0.0000% at (0,0)
frm 39: diff from frm 40 = 0.0075% at (0,0)
frm 40: diff from frm 41 = 0.0000% at (0,0)
frm 41: diff from frm 42 = 0.0162% at (128,0)
frm 42: diff from frm 43 = 0.0000% at (0,0)
frm 43: diff from frm 44 = 0.0224% at (256,0)
frm 44: diff from frm 45 = 0.0000% at (0,0)
frm 45: diff from frm 46 = 0.0204% at (256,0)
frm 46: diff from frm 47 = 0.0000% at (0,0)
frm 47: diff from frm 48 = 0.0220% at (0,0)
frm 48: diff from frm 49 = 0.0000% at (0,0)
frm 49: diff from frm 50 = 0.0112% at (128,0)
frm 50: diff from frm 51 = 0.0000% at (0,0)
frm 51: diff from frm 52 = 0.0249% at (256,0)
frm 52: diff from frm 53 = 0.0000% at (0,0)
frm 53: diff from frm 54 = 0.0175% at (256,0)
frm 54: diff from frm 55 = 0.0000% at (0,0)
frm 55: diff from frm 56 = 0.0162% at (256,0)
frm 56: diff from frm 57 = 0.0000% at (0,0)
frm 57: diff from frm 58 = 0.0087% at (192,0)
frm 58: diff from frm 59 = 0.0000% at (0,0)
frm 59: diff from frm 60 = 0.0129% at (0,0)
frm 60: diff from frm 61 = 0.0000% at (0,0)

So, I think we're set now. This wouldn't have been possible without Quaz spending a lot of time examining videos and read-outs, so kudos to him for helping out and lending his incredible eye for detail to this endevour.

 

[Quaz51]

yeah good

i analyze your GTA X360 video send to me and i understand how use the Vsync, the Vsync is always enable except when the frame is ready less 10% before the second VBL or less 10% after the second VBL, all the other case the Vsync seem enable (so the tearing appaer only around the second VBL, not around the one or third VBL, and only on top and down of the frame, not very visible)
it's almost impossible in this case to have two successive torn frame, you can use the method for count torn frame in GTA X360

 

[grandmaster]

Yes, we've spent a lot of time looking at various GTA captures too. Suffice to say that the revised results are not very different at all from the Eurogamer originals. There appears to be a problem with the PS3 capture of Test4 though so I want to recapture that section.

i analyze your GTA X360 video send to me and i understand how use the Vsync, the Vsync is always enable except when the frame is ready less 10% before the second VBL or less 10% after the second VBL, all the other case the Vsync seem enable (so the tearing appaer only around the second VBL, not around the one or third VBL, and only on top and down of the frame, not very visible)
it's almost impossible in this case to have two successive torn frame, you can use the method for count torn frame in GTA X360

So to clarify, the screen tear is very rare, and on many displays has a high probability of appearing in the overscan area.



Not only that, but as you will not get two torn frames in a row, the affect stays on-screen for 1/60th of a second only. I'd say that it's handled very intelligently.

 

[AlNom]

Hm... so it sounds a bit like how Gears of War enables/disables V-sync.

 

[rekator]

Great work and great thanks Grandmaster and Quaz51!

 

[grandmaster]

I've revised the Eurogamer GTA IV tests. The original article has been updated accordingly, and changes explained on the Editor's blog.

The increased precision in the new revision of the scanner makes these results definitive, but weirdly the old version wasn't a million miles away. The less precise tolerance level served to factor out a lot of the v-lock effect. Also bear in mind Quaz51's previous post about the way v-lock is handled on 360, and the minor impact screen tear has on overall image quality on this specific game.

Once again, to avoid confusion, please note that the torn frame percentage is derived from the original 60fps capture, not the unique frame count calculation which compensates for them.

For those looking for 100% like-for-like comparisons, Test1 is the full intro sequence, while Test3 is an indoor cut-scene with what appears to be no external lighting. Test4 still looks rather incongruous. For the revised test I actually recaptured it, yet the overall result has hardly altered.

GTA IV Frame Rate Tests

Test1
Old 360: 31.990fps
Old PS3: 26.460fps
New 360: 31.627fps (3.77% torn frames)
New PS3: 26.504fps

Test2
Old 360: 28.624fps
Old PS3: 23.452fps
New 360: 29.233fps (4.26% torn frames)
New PS3: 23.989fps

Test3
Old 360: 35.252fps
Old PS3: 29.041fps
New 360: 35.662fps (2.37% torn frames)
New PS3: 29.523fps

Test4
Old 360: 26.076fps
Old PS3: 26.081fps
New 360: 26.015fps (6.07% torn frames)
New PS3: 25.803fps

Test5
Old 360: 26.712fps
Old PS3: 23.781fps
New 360: 26.507fps (4.9% torn frames)
New PS3: 23.781fps

Test6
Old 360: 33.798fps
Old PS3: 28.313fps
New 360: 33.713fps (2.35% torn frames)
New PS3: 28.313fps


Thanks to everyone here who has taken part in the discussion and helped spur us on to get the methodology sorted.

 

[goonergaz]

IMHO Tests 4 and 5 show that you should maybe add a disclaimer that resurts are within a tolerance of approx 5% either way...therefore meaning that comparing any results directly COULD have a difference of up to 10% (admited that's a worst case scenario, but I think it's be worth mentioning).

Nice to see it's now a 'fairer' method of measuring tho

 

[habbe]

i still don't get how you measure torn frames ?. If you only counting duped frames.
It would work if the torn frame is presented again in next frame with the same first line.

Does every engine render from the left to the right and downwards ?.