Frame Rate Analysis Thread (Simple Rules Post #2)

[Quaz51]

i still don't get how you measure torn frames ?. If you only counting duped frames.

the fps-detector count duplicate frame, compare current frame with next. when you use just one line test comparison you find the true framerate even if tearing. fullscreen test comparison on the same sequence count all torn frame like a new frame not like a duplicate frame. if you do a fullscreen test on the same sequence then for each torn frame you have -1 duplicate frame on the previous result. compare duplicate frame in both test, the difference is the number of torn frame
it work only if there never are two successive torn frame (at 60hz) on the sequence, so counting torn frame don't work on 1VBL game (but counting framerate work on all game, it's the more important. the exact number of torn frame is just a bonus)

 

[RobertR1]

grandmaster and/or quazmaster,

At what percentage of "torn frames" would you say has a visible and negative effect on gameplay. 5% 10% 20?

 

[Mintmaster]

IMHO Tests 4 and 5 show that you should maybe add a disclaimer that resurts are within a tolerance of approx 5% either way...therefore meaning that comparing any results directly COULD have a difference of up to 10%

How do you come up with that conclusion? Just because there's ~5% torn frames doesn't mean that the framerate measurements are getting affected.

At what percentage of "torn frames" would you say has a visible and negative effect on gameplay. 5% 10% 20?

From what they're saying, it doesn't seem to be that simple. The 360 is apparently 'cheating' on its v-sync so as to minimize framerate impact while retaining most of the benefits. Lots of those tears are hard to notice because they're near the edges or even off the screen entirely (but still detected by the program).

Of course the smart thing to do would be just use 0.7% of the RAM to enable triple buffering.

 

[Mintmaster]

i still don't get how you measure torn frames ?. If you only counting duped frames.

They're not, though.

Think of the captured video as a stream of lines, not a stream of frames. You can compare the current line with one that is 720 lines ahead, and when it's different you know a frame transition occurred. Now you make the latter your current line and repeat.

Count all the frame transitions, divide by the total time, and there you go. The only problem is when framerate is above 60 fps and v-sync is disabled. It would be rather silly for a dev to do that, though...

Good work, grandmaster and Quaz! I especially like your analysis of what was happening on the 360.

EDIT: Seems I made a mistake. All grandmaster is doing is comparing the middle line of one frame to the next. Just as good for this length of video, though...

 

[Quaz51]

The only problem is when framerate is above 60 fps and v-sync is disabled. It would be rather silly for a dev to do that, though...

yes, indeed, in theory this case don't exist

 

[Pete]

OT, but Quaz is 'infamous' now? I thought Americans were supposed to abuse English, not y'all!

And thanks for the work you've done and the time you've spent explaining it here, gm! I'd thank Quaz again, too, but I don't want to encourage unsavory characters. =P

 

[goonergaz]

How do you come up with that conclusion? Just because there's ~5% torn frames doesn't mean that the framerate measurements are getting affected.

OK, so all this is doing is making the measurements "more accurate" - we will never get a like for like comparison result because v-lock is scewing it - we can only assume the screen tear on PS3 was more evident? Also you can't accurately recreate 'gameplay' due to uncontrollable things (like how many cars/peds etc) - the results show that (IMHO).

I suggest that if "ingame" tests were ran 3 times and an average taken that would reflect a more accurate and fair score as no matter how hard you try to recreate the same tests it's simply impossible. 3DMark/Doom3 tests were accurate as it ran a scripted event, it's a shame games don't include that - other than cut scenes...also I wonder how much difference HDD type/condition makes on PS3 performance.

Be also interesting to measure a scene with rain also - apparently that's when tearing is most evident. Were any of the tests during rain? I also wonder why test 4 is so close yet the others so different?

 

[catisfit]

OK, so all this is doing is making the measurements "more accurate" - we will never get a like for like comparison result because v-lock is scewing it - we can only assume the screen tear on PS3 was more evident? Also you can't accurately recreate 'gameplay' due to uncontrollable things (like how many cars/peds etc) - the results show that (IMHO).

I suggest that if "ingame" tests were ran 3 times and an average taken that would reflect a more accurate and fair score as no matter how hard you try to recreate the same tests it's simply impossible. 3DMark/Doom3 tests were accurate as it ran a scripted event, it's a shame games don't include that - other than cut scenes...also I wonder how much difference HDD type/condition makes on PS3 performance.

Be also interesting to measure a scene with rain also - apparently that's when tearing is most evident. Were any of the tests during rain? I also wonder why test 4 is so close yet the others so different?

I also think that a straight average isn't as useful as high/median/low values for the framerate, e.g. the following exaggerated example:

40 30 0 0 40 60 20 0 30 40 0 40 = 25fps
25 25 25 25 25 25 25 25 25 25 25 25 = 25fps

Would you prefer the one with 2x1s pauses and 1x2s pause, or the consistent 25fps?!

 

[goonergaz]

I also think that a straight average isn't as useful as high/median/low values for the framerate, e.g. the following exaggerated example:

40 30 0 0 40 60 20 0 30 40 0 40 = 25fps
25 25 25 25 25 25 25 25 25 25 25 25 = 25fps

Would you prefer the one with 2x1s pauses and 1x2s pause, or the consistent 25fps?!

good shout...and absolutely spot on IMO

 

[grandmaster]

@eb: If you want to see the material being tested, the 360 versions are available as streaming video in the original feature. You'll see that Test5 in particular is nothing that strenuous - indeed, the amount of people, the amount of cars, everything is heavily scripted there and is basically the same cross-platform. Test4 is also very close content-wise indeed. There's only so many people that can fit on that station platform.

As a last word on the GTA IV saga before I delete the majority of the captures, and in an effort to keep this thread on-topic, I combined a large majority of my PS3 and 360 captures into compilation clips and measured them.

This is about 80 minutes of video for each game, spread over the same 20 missions - same cut-scenes and same basic gameplay. A rough calculation for sure, but with over a quarter of a million frames measured on each console, an indication of an overall trend. At least I hope so as it occupied my PC for around four hours to make the calculations.

Xbox 360: 30.264fps (vs 30.456fps average for Tests1-6)
PS3: 26.521fps (vs 26.049 average in Tests1-6)

Perhaps this would be an alternative if you want to factor in an overall tolerance.

I also think that a straight average isn't as useful as high/median/low values for the framerate

Excellent idea, I've had the same thoughts. The detector pumps out data on how much every frame is different from the last (as seen by the Race Driver GRID sample above) so there's a lot more meaningful detail we can derive from the analysis.

It won't be immediate but I'm working on it. Suggestions on processing the data and how you want it presented are welcome.

 

[catisfit]

As a last word on the GTA IV saga before I delete the majority of the captures, and in an effort to keep this thread on-topic, I combined a large majority of my PS3 and 360 captures into compilation clips and measured them.

This is about 80 minutes of video for each game, spread over the same 20 missions - same cut-scenes and same basic gameplay. A rough calculation for sure, but with over a quarter of a million frames measured on each console, an indication of an overall trend. At least I hope so as it occupied my PC for around four hours to make the calculations.

Xbox 360: 30.264fps (vs 30.456fps average for Tests1-6)
PS3: 26.521fps (vs 26.049 average in Tests1-6)

Good work

 

[Shifty Geezer]

It won't be immediate but I'm working on it. Suggestions on processing the data and how you want it presented are welcome.

A graph. You could plot time on screen for each frame, so a long flat line would show consistency, and the height would show framerate. Or a histogram with a narrow band representing lower variation and a broad band showing lots of fluctuations. Either will provide an at-a-glance comparison of performance fluctuations and be overall better understood and more applicable than some different averaging values.

 

[Quaz51]

Good work

yes

 

[grandmaster]

A graph. You could plot time on screen for each frame, so a long flat line would show consistency, and the height would show framerate. Or a histogram with a narrow band representing lower variation and a broad band showing lots of fluctuations. Either will provide an at-a-glance comparison of performance fluctuations and be overall better understood and more applicable than some different averaging values.

Yes, first thoughts are a graph with y axis = frames per second, x axis = time. I'm wondering if screen tear could be represented by the line of the graph being broken somehow.

 

[goonergaz]

yes

only good?

 

[mrcorbo]

Yes, first thoughts are a graph with y axis = frames per second, x axis = time. I'm wondering if screen tear could be represented by the line of the graph being broken somehow.

Visually, I think individual torn frames could be represented by colored vertical lines in the line of the graph.

 

[Mintmaster]

OK, so all this is doing is making the measurements "more accurate" - we will never get a like for like comparison result because v-lock is scewing it - we can only assume the screen tear on PS3 was more evident? Also you can't accurately recreate 'gameplay' due to uncontrollable things (like how many cars/peds etc) - the results show that (IMHO).

There was no screen tear on PS3. It has the traditional v-sync in this game.

The whole point of these tests is to compare the gameplay experience that you get from of different versions of a game or maybe even different games, not the GPUs. These comparisons can't be used to judge the difference in hardware speed because v-sync will always give slower results; moreover, the effect isn't consistent. Depending on the exact sequence of rendering times, it can either mask or accentuate the speed differences between platforms.

I suggest that if "ingame" tests were ran 3 times and an average taken that would reflect a more accurate and fair score as no matter how hard you try to recreate the same tests it's simply impossible.

Sure, but you still don't have any reason to say that the variance is +/- 5%. The GTA4 tests were cut-scenes with only minor differences, the GRID tests have little variation, and for other in-game tests, a reasonable length (say 1-2 minutes) is enough to even out these variances for the most part.

Be also interesting to measure a scene with rain also - apparently that's when tearing is most evident. Were any of the tests during rain? I also wonder why test 4 is so close yet the others so different?

Good points about the rain, as at least one member is saying there is some correlation between weather and tearing. The test 4 parity is interesting, but it should be noted that tests 4 and 5 (which also shows 360 with a smaller lead) are the shortest clips tested at ~15s.

 

[grandmaster]

The shortness of the GTA cut-scenes and the resultant drop in frame rate are an excellent reason for the graphs being talked about. With a bit of work the graph could run below the video.

Next up is going to be the "does Xenos scaling affect performance" test. I've already measured Call of Duty 4's chopper/boat/rain intro sequence and that is rock solid 60fps at both 1080p and 720p. Quaz51 suggested that Dead Rising would be a good game to test. A 720p60 game with screen-tear would be a good candidate.

I found some Call of Duty 4 (360) gameplay clips in my archive. Results are:

Clip #1: 1029 frames, 0 duplicates, 60fps
Clip #2: 1453 frames, 7 dupes, 59.711fps
Clip #3: 1241 frames, 10 dupes, 59.516fps
Clip #4: 1484 frames, 1 dupe, 59.94fps
Clip #5: 1393 frames 1 dupe, 59.957fps
Clip #6: 1660 frames, 0 dupes, 60fps
Clip #7: 1313 frames, 99 dupes, 55.476fps
Clip #8: 1686 frames, 0 dupes, 60fps
Clip #9: 1283 frames, 1 dupe, 59.95fps
Clip #10: 1863 frames, 8 dupes, 59.742fps
Clip #11: 1497 frames, 10 dupes, 59.599fps

Clip #7 is interesting - it's from the bit where you rescue the helicopter pilot while gunfire is raining in from across the street. The action in this game is so intense, I can't spot the dropped frames.

Bearing in mind that the CoD4 engine is being re-used for CoD5 as well as the new Bond games, I'm sourcing the PS3 version for some comparison measurements. According to Quaz51, the PS3 version has some screen tear. The duped frames above are all exact duplicates, there's not a hint of screen-tear in any of these clips, nor the chopper intro or the car tour "coup" sequence.

 

[Quaz51]

Quaz51 suggested that Dead Rising would be a good game to test

not for upscale test, just for understand his strange Vsync

According to Quaz51, the PS3 version has some screen tear.

i don't remember if there is tearing but they are slowdowns

 

[pjbliverpool]

The shortness of the GTA cut-scenes and the resultant drop in frame rate are an excellent reason for the graphs being talked about. With a bit of work the graph could run below the video.

Next up is going to be the "does Xenos scaling affect performance" test. I've already measured Call of Duty 4's chopper/boat/rain intro sequence and that is rock solid 60fps at both 1080p and 720p. Quaz51 suggested that Dead Rising would be a good game to test. A 720p60 game with screen-tear would be a good candidate.

I found some Call of Duty 4 (360) gameplay clips in my archive. Results are:

Clip #1: 1029 frames, 0 duplicates, 60fps
Clip #2: 1453 frames, 7 dupes, 59.711fps
Clip #3: 1241 frames, 10 dupes, 59.516fps
Clip #4: 1484 frames, 1 dupe, 59.94fps
Clip #5: 1393 frames 1 dupe, 59.957fps
Clip #6: 1660 frames, 0 dupes, 60fps
Clip #7: 1313 frames, 99 dupes, 55.476fps
Clip #8: 1686 frames, 0 dupes, 60fps
Clip #9: 1283 frames, 1 dupe, 59.95fps
Clip #10: 1863 frames, 8 dupes, 59.742fps
Clip #11: 1497 frames, 10 dupes, 59.599fps

Clip #7 is interesting - it's from the bit where you rescue the helicopter pilot while gunfire is raining in from across the street. The action in this game is so intense, I can't spot the dropped frames.

They are very interesting results. I have to say i'm suprised the 360 is sticking to 60fps so closely.

When you mention the chopper/boat/rain intro sequence, is that were you are actually inside the cockpit flying around the boat? And is there any camera movement during the clip? I.e. are you looking around the cockpit andd over the boat throughout or does the camera just stay at its starting position?