Frame Rate Analysis Thread (Simple Rules Post #2)

Status
Not open for further replies.
Further to the FPS measurements in the Grand Theft Auto Eurogamer face-off, this thread is all about discussing methodology for establishing frame rate of any given video clip, and posting results.

The clips in question are 720p60 24-bit lossless videos captured from the HDMI ports of Xbox 360 and PlayStation.

Current methodology is as described in the Race Driver GRID (demo) test.

Results

1. Race Driver GRID (demo, Xbox 360/PS3)
2. Grand Theft Auto IV (Xbox 360/PS3)
3. Call of Duty 4 (Xbox 360)
4. Metal Gear Solid 4 (PS3)
5. Star Wars: The Force Unleashed (demo, Xbox 360)
6. Mercenaries II: World in Flames (Xbox 360/PS3)
 
Last edited by a moderator:
Mod Helm of (near) Invincibility

I want to make it clear that this is not a thread to voice console bias nor to make snide remarks against developers nor for some-Defense Force to show up. So please leave your "political" associations and comments out of this thread.

This is a thread to analyze frame rates. Because none of us has a built-in PerfHUD or PIX (please get some help if you think you do ;) ), it would be interesting to discuss possible reasons for a particular observed performance specific to a game in general or to the scene in question - whether it be shader effects, rendering methods, CPU issues, V-sync, etc



 
So is gradmaster eurogamer staff? Because they're using the same method he is.

It would be awesome to start getting frame rate data on top of our resolution data, hope this screen tearing thing doesn't throw it off.
 
I do the tests for Eurogamer. In fact all of their homegrown vids - not just the comparison feature assets - are produced using my games-specific capture technology.

It's fairly easy to spot a game with v-lock, so those direct comparisons tests will be accurate. Games like GTA where the v-lock is negligable (at least on the test clips chosen) will also be valuable.

But with code like the Race Driver GRID demo, all bets are off. It's obvious a 30fps game, and the more I play the 360 version, the more I'm convinced it's v-locked during the races. But when you see PS3 frame rates at 40-50fps you're also getting some idea of how bad the screen tear is.
 
So a higher framerate is normally good, but tearing is bad. We knew that, but it also means that the combination of the two is what is really important. To talk about this a little more intelligently, we need more.

I think that if you have a programmer who can detect fps and screan tearing, then he can also probably do the following:

Make a graph showing the time on the x-axis, the framerate on the y-axis, and add small vertical stripes to the dots for each frame that tears. This will give all the information needed to talk about framerates and screen tearing precisely and informedly. You can see for which (type of) scenes the framerate spikes, what situations cause the most tearing, and so on. And because the context here is comparing versions, you can overlay the two graphs nicely, and then fairly easily explain what differences occur when and why. It is also important to realise whether or not it is more pleasant to have a framerate locked at 30fps, or a framerate that can move from anywhere between 30-50 fps for instance, because otherwise god forbid programmers will choose the latter option because this will result in higher fps in Eurogamer comparison features. ;)

Seriously, if you could achieve the above graph, that would be absolutely fantastic, and I don't see why that wouldn't be possible?
 
The point is that what we are doing is counting unique frames. I've no idea how to process torn ones. In a game like GTA IV, the torn frames are rare and don't have so much impact on the sample.

In theory you could perhaps code around that and count such frames as duplicates, but what about if the video only actually has motion in, say, the top part of the screen? I'm not sure how the program could differentiate between that and v-lock tear.

And now, on Race Driver obviously we're seeing torn frame after torn frame. It just makes the job of detection even more difficult to implement..

I'm open to ideas on how the scanner could be upgraded to cope with both eventualities.


By the way, if there are any frame rate requests, I can give it a go. Haze is a solid 30fps by the way.
 
Last edited by a moderator:
surely such high values in frame rate would suggest they could increase the detail on the ps3 version. Still, we see the ps3 as the worse version in the comparative threads, in detail and AA.

Its hard to make sense out of multi-platform titles and I sure as hell don't understand it
 
You don't get it. The high value of the frame rates isn't showing that the PS3 version is better, faster, or capable of more detail. It's that it's running with no v-lock, can't achieve a solid frame rate and is displaying a hell of a lot of torn frames.
 
You don't get it. The high value of the frame rates isn't showing that the PS3 version is better, faster, or capable of more detail. It's that it's running with no v-lock, can't achieve a solid frame rate and is displaying a hell of a lot of torn frames.

Are you saying something along the lines of 2 torn frames = 1 non torn frames in performance terms?

i.e. more frames is not better if 50% of them are torn because it represents less overall work?

Great thread btw, i'm really looking forward to seeing what it comes out with. I would love to see CoD4's framerates. Both consoles supposedly run at a locked 60fps but i'm dubious.
 
You don't get it. The high value of the frame rates isn't showing that the PS3 version is better, faster, or capable of more detail. It's that it's running with no v-lock, can't achieve a solid frame rate and is displaying a hell of a lot of torn frames.

Or it could be said that torn frames are preventing you from achieving a solid measure of frame rates.

I really like the idea of this thread but only 8 posts in you are making conclusions from measurements you already believe to be faulty.
 
You don't get it. The high value of the frame rates isn't showing that the PS3 version is better, faster, or capable of more detail. It's that it's running with no v-lock, can't achieve a solid frame rate and is displaying a hell of a lot of torn frames.

what it shows us is the measurements are not 100% accurate - no matter how 'minimal' the tear is on GTA (or big on GRID) it will affect the result...so therefore it can't be used to compare performances across platforms (apples to oranges and all that)
 
I'm a bit confused, what I'm reading here is that the PS3 version isn't able to maintain 30fps and since it isn't v-locked there's tearing and it screws your framerate counting, making it appear as if it was much higher OR the PS3 version, since it has no v-lock, goes above 30, causes tearing and this would be fixable by locking?

I'm guessing the 1st option... right?
 
I'm a bit confused, what I'm reading here is that the PS3 version isn't able to maintain 30fps and since it isn't v-locked there's tearing and it screws your framerate counting, making it appear as if it was much higher OR the PS3 version, since it has no v-lock, goes above 30, causes tearing and this would be fixable by locking?
If the tearing was fixable by enabling v-sync, they'd have to be kind of retarded not to do it don't you think? I'm not saying that isn't the case, but nobody wants to put their game out and not have it look its best on a demo.

I like the idea of the thread as it's really the first time we've had a definitive test for frame rate but after the confusion regarding the maiden post I'm not sure how helpful it'll be unless you can rework your algorithm to better make sense of torn frames. The first frame should be the standard frame height, so it seems like you could just check each "new frame" that your analysis tool came up with and if it's shortened then decrement the counted frame total by one as otherwise when you counted the next partial frame (the other section of the torn "screen") you'd be counting the frame twice and artificially boosting the frame rate total. Add another counter for the number of shortened frames and you can even come up with a percentage of time that the game is tearing.
 
If the tearing was fixable by enabling v-sync, they'd have to be kind of retarded not to do it don't you think? I'm not saying that isn't the case, but nobody wants to put their game out and not have it look its best on a demo.

Yeah I know, if the framerate was higher it would be fixable by v-locking, and it wouldn't make sense not to do it. It's just that some of the comments in the thread were making me wonder if I was wrong. :???:
 
If the tearing was fixable by enabling v-sync, they'd have to be kind of retarded not to do it don't you think? I'm not saying that isn't the case, but nobody wants to put their game out and not have it look its best on a demo.


if this is the case - out of interest - why was the retail of GTA not fixed? esp. as it (apparently) doesn't happen much...surely locking it wouldn't affect the f/r much so therefore they should have locked it...or conversely if the PS3 version has the poorer f/r why not unlock it!?

just asking from an uneducated POV
 
if this is the case - out of interest - why was the retail of GTA not fixed? esp. as it (apparently) doesn't happen much...surely locking it wouldn't affect the f/r much so therefore they should have locked it...or conversely if the PS3 version has the poorer f/r why not unlock it!?
You'd have to ask the developer to know why they did or didn't do something on any particular game. In the case of GTA maybe they didn't think it was enough of a problem to spend the money fixing it. It's not like screen tearing is the most obvious or even all that common a visual flaw on the 360 version.

assen said:
The torn captured frames are probably not shortened, but contain a slice of the previous and a slice of the next frame.
I guess it depends on how grandmaster's tool is determining what a unique frame is and how it's counting them. If it's not counting a torn frame as 2 unique frames then never mind. :)
 
Assen is right on the money. A torn frame looks like this:



The trick is going to changing the tool to note this as a torn frame, not as a unique one. I've had an idea how to do that, but on games where there are two torn frames in a row (Moto GP 06 I'm looking at you), it's going to be a problem.

I was wrong about Race Driver though. It doesn't appear to be displaying two torn frames in a row. You'll have a proper frame, sometimes two, then a torn one.
 
grandmaster said:
I was wrong about Race Driver though. It doesn't appear to be displaying two torn frames in a row. You'll have a proper frame, sometimes two, then a torn one.
The only way you could get non-stop torn frames would be if it was running above 60 or at least near 60 (to have many consecutive ones).

Anyway, so long as the game is not over 60, it should be possible to exactly measure fps, the torn frames should not be discarded, they should be measured as well - I posted about this in the other thread. Maybe I should copy paste it from there?
If you need help with heuristic for detecting and measuring height of torn frames, let us know, it shouldn't be too hard.
 
Last edited by a moderator:
Status
Not open for further replies.
Back
Top