Frame Rate Analysis Thread (Simple Rules Post #2)

Discussion in 'Console Technology' started by grandmaster, May 9, 2008.

Thread Status:
Not open for further replies.
  1. grandmaster

    Veteran

    Joined:
    Feb 6, 2007
    Messages:
    1,159
    Likes Received:
    0
    Further to the FPS measurements in the Grand Theft Auto Eurogamer face-off, this thread is all about discussing methodology for establishing frame rate of any given video clip, and posting results.

    The clips in question are 720p60 24-bit lossless videos captured from the HDMI ports of Xbox 360 and PlayStation.

    Current methodology is as described in the Race Driver GRID (demo) test.

    Results

    1. Race Driver GRID (demo, Xbox 360/PS3)
    2. Grand Theft Auto IV (Xbox 360/PS3)
    3. Call of Duty 4 (Xbox 360)
    4. Metal Gear Solid 4 (PS3)
    5. Star Wars: The Force Unleashed (demo, Xbox 360)
    6. Mercenaries II: World in Flames (Xbox 360/PS3)
     
    #1 grandmaster, May 9, 2008
    Last edited by a moderator: Sep 25, 2008
  2. TheAlSpark

    TheAlSpark Moderator
    Moderator Legend

    Joined:
    Feb 29, 2004
    Messages:
    21,705
    Likes Received:
    7,337
    Location:
    ಠ_ಠ
    Mod Helm of (near) Invincibility

    I want to make it clear that this is not a thread to voice console bias nor to make snide remarks against developers nor for some-Defense Force to show up. So please leave your "political" associations and comments out of this thread.

    This is a thread to analyze frame rates. Because none of us has a built-in PerfHUD or PIX (please get some help if you think you do ;) ), it would be interesting to discuss possible reasons for a particular observed performance specific to a game in general or to the scene in question - whether it be shader effects, rendering methods, CPU issues, V-sync, etc



     
  3. Rangers

    Legend

    Joined:
    Aug 4, 2006
    Messages:
    12,607
    Likes Received:
    1,418
    So is gradmaster eurogamer staff? Because they're using the same method he is.

    It would be awesome to start getting frame rate data on top of our resolution data, hope this screen tearing thing doesn't throw it off.
     
  4. grandmaster

    Veteran

    Joined:
    Feb 6, 2007
    Messages:
    1,159
    Likes Received:
    0
    I do the tests for Eurogamer. In fact all of their homegrown vids - not just the comparison feature assets - are produced using my games-specific capture technology.

    It's fairly easy to spot a game with v-lock, so those direct comparisons tests will be accurate. Games like GTA where the v-lock is negligable (at least on the test clips chosen) will also be valuable.

    But with code like the Race Driver GRID demo, all bets are off. It's obvious a 30fps game, and the more I play the 360 version, the more I'm convinced it's v-locked during the races. But when you see PS3 frame rates at 40-50fps you're also getting some idea of how bad the screen tear is.
     
  5. Arwin

    Arwin Now Officially a Top 10 Poster
    Moderator Legend

    Joined:
    May 17, 2006
    Messages:
    18,063
    Likes Received:
    1,660
    Location:
    Maastricht, The Netherlands
    So a higher framerate is normally good, but tearing is bad. We knew that, but it also means that the combination of the two is what is really important. To talk about this a little more intelligently, we need more.

    I think that if you have a programmer who can detect fps and screan tearing, then he can also probably do the following:

    Make a graph showing the time on the x-axis, the framerate on the y-axis, and add small vertical stripes to the dots for each frame that tears. This will give all the information needed to talk about framerates and screen tearing precisely and informedly. You can see for which (type of) scenes the framerate spikes, what situations cause the most tearing, and so on. And because the context here is comparing versions, you can overlay the two graphs nicely, and then fairly easily explain what differences occur when and why. It is also important to realise whether or not it is more pleasant to have a framerate locked at 30fps, or a framerate that can move from anywhere between 30-50 fps for instance, because otherwise god forbid programmers will choose the latter option because this will result in higher fps in Eurogamer comparison features. ;)

    Seriously, if you could achieve the above graph, that would be absolutely fantastic, and I don't see why that wouldn't be possible?
     
  6. grandmaster

    Veteran

    Joined:
    Feb 6, 2007
    Messages:
    1,159
    Likes Received:
    0
    The point is that what we are doing is counting unique frames. I've no idea how to process torn ones. In a game like GTA IV, the torn frames are rare and don't have so much impact on the sample.

    In theory you could perhaps code around that and count such frames as duplicates, but what about if the video only actually has motion in, say, the top part of the screen? I'm not sure how the program could differentiate between that and v-lock tear.

    And now, on Race Driver obviously we're seeing torn frame after torn frame. It just makes the job of detection even more difficult to implement..

    I'm open to ideas on how the scanner could be upgraded to cope with both eventualities.


    By the way, if there are any frame rate requests, I can give it a go. Haze is a solid 30fps by the way.
     
    #6 grandmaster, May 9, 2008
    Last edited by a moderator: May 9, 2008
  7. dskneo

    Regular

    Joined:
    Jul 25, 2005
    Messages:
    531
    Likes Received:
    45
    surely such high values in frame rate would suggest they could increase the detail on the ps3 version. Still, we see the ps3 as the worse version in the comparative threads, in detail and AA.

    Its hard to make sense out of multi-platform titles and I sure as hell don't understand it
     
  8. grandmaster

    Veteran

    Joined:
    Feb 6, 2007
    Messages:
    1,159
    Likes Received:
    0
    You don't get it. The high value of the frame rates isn't showing that the PS3 version is better, faster, or capable of more detail. It's that it's running with no v-lock, can't achieve a solid frame rate and is displaying a hell of a lot of torn frames.
     
  9. pjbliverpool

    pjbliverpool B3D Scallywag
    Legend

    Joined:
    May 8, 2005
    Messages:
    7,829
    Likes Received:
    1,142
    Location:
    Guess...
    Are you saying something along the lines of 2 torn frames = 1 non torn frames in performance terms?

    i.e. more frames is not better if 50% of them are torn because it represents less overall work?

    Great thread btw, i'm really looking forward to seeing what it comes out with. I would love to see CoD4's framerates. Both consoles supposedly run at a locked 60fps but i'm dubious.
     
  10. upnorthsox

    Veteran

    Joined:
    May 7, 2008
    Messages:
    2,102
    Likes Received:
    378
    Or it could be said that torn frames are preventing you from achieving a solid measure of frame rates.

    I really like the idea of this thread but only 8 posts in you are making conclusions from measurements you already believe to be faulty.
     
  11. goonergaz

    Veteran

    Joined:
    Jun 3, 2005
    Messages:
    3,601
    Likes Received:
    1,041
    what it shows us is the measurements are not 100% accurate - no matter how 'minimal' the tear is on GTA (or big on GRID) it will affect the result...so therefore it can't be used to compare performances across platforms (apples to oranges and all that)
     
  12. FirewalkR

    Regular

    Joined:
    Jul 13, 2007
    Messages:
    259
    Likes Received:
    0
    I'm a bit confused, what I'm reading here is that the PS3 version isn't able to maintain 30fps and since it isn't v-locked there's tearing and it screws your framerate counting, making it appear as if it was much higher OR the PS3 version, since it has no v-lock, goes above 30, causes tearing and this would be fixable by locking?

    I'm guessing the 1st option... right?
     
  13. Pugger

    Regular

    Joined:
    Dec 8, 2004
    Messages:
    419
    Likes Received:
    2
    Grandmaster, is that Richard Ledbetter(sp)?
     
  14. chachi

    Newcomer

    Joined:
    Sep 15, 2004
    Messages:
    120
    Likes Received:
    3
    If the tearing was fixable by enabling v-sync, they'd have to be kind of retarded not to do it don't you think? I'm not saying that isn't the case, but nobody wants to put their game out and not have it look its best on a demo.

    I like the idea of the thread as it's really the first time we've had a definitive test for frame rate but after the confusion regarding the maiden post I'm not sure how helpful it'll be unless you can rework your algorithm to better make sense of torn frames. The first frame should be the standard frame height, so it seems like you could just check each "new frame" that your analysis tool came up with and if it's shortened then decrement the counted frame total by one as otherwise when you counted the next partial frame (the other section of the torn "screen") you'd be counting the frame twice and artificially boosting the frame rate total. Add another counter for the number of shortened frames and you can even come up with a percentage of time that the game is tearing.
     
  15. FirewalkR

    Regular

    Joined:
    Jul 13, 2007
    Messages:
    259
    Likes Received:
    0
    Yeah I know, if the framerate was higher it would be fixable by v-locking, and it wouldn't make sense not to do it. It's just that some of the comments in the thread were making me wonder if I was wrong. :???:
     
  16. assen

    Veteran

    Joined:
    May 21, 2003
    Messages:
    1,377
    Likes Received:
    19
    Location:
    Skirts of Vitosha
    The torn captured frames are probably not shortened, but contain a slice of the previous and a slice of the next frame.
     
  17. goonergaz

    Veteran

    Joined:
    Jun 3, 2005
    Messages:
    3,601
    Likes Received:
    1,041

    if this is the case - out of interest - why was the retail of GTA not fixed? esp. as it (apparently) doesn't happen much...surely locking it wouldn't affect the f/r much so therefore they should have locked it...or conversely if the PS3 version has the poorer f/r why not unlock it!?

    just asking from an uneducated POV
     
  18. chachi

    Newcomer

    Joined:
    Sep 15, 2004
    Messages:
    120
    Likes Received:
    3
    You'd have to ask the developer to know why they did or didn't do something on any particular game. In the case of GTA maybe they didn't think it was enough of a problem to spend the money fixing it. It's not like screen tearing is the most obvious or even all that common a visual flaw on the 360 version.

    I guess it depends on how grandmaster's tool is determining what a unique frame is and how it's counting them. If it's not counting a torn frame as 2 unique frames then never mind. :)
     
  19. grandmaster

    Veteran

    Joined:
    Feb 6, 2007
    Messages:
    1,159
    Likes Received:
    0
    Assen is right on the money. A torn frame looks like this:

    [​IMG]

    The trick is going to changing the tool to note this as a torn frame, not as a unique one. I've had an idea how to do that, but on games where there are two torn frames in a row (Moto GP 06 I'm looking at you), it's going to be a problem.

    I was wrong about Race Driver though. It doesn't appear to be displaying two torn frames in a row. You'll have a proper frame, sometimes two, then a torn one.
     
  20. Fafalada

    Veteran

    Joined:
    Feb 8, 2002
    Messages:
    2,773
    Likes Received:
    49
    The only way you could get non-stop torn frames would be if it was running above 60 or at least near 60 (to have many consecutive ones).

    Anyway, so long as the game is not over 60, it should be possible to exactly measure fps, the torn frames should not be discarded, they should be measured as well - I posted about this in the other thread. Maybe I should copy paste it from there?
    If you need help with heuristic for detecting and measuring height of torn frames, let us know, it shouldn't be too hard.
     
    #20 Fafalada, May 9, 2008
    Last edited by a moderator: May 9, 2008
Loading...
Thread Status:
Not open for further replies.

Share This Page

  • About Us

    Beyond3D has been around for over a decade and prides itself on being the best place on the web for in-depth, technically-driven discussion and analysis of 3D graphics hardware. If you love pixels and transistors, you've come to the right place!

    Beyond3D is proudly published by GPU Tools Ltd.
Loading...