Realtime frame interpolation upscaling 30fps to 60fps

Well... the best case would be "no change" from frame to frame, and the worst case is everything is different (scene change)... you can't really program a dynamic game with those variables, unless you always take the worst case in mind and don't benefit from it at all (well... you might get better framerate low frame change scenes, but in exactly those cases, the framerate is less important^^)
 
It's better to reproject every frame partially (the sections with low error metric) and refresh (render fully) only parts that are needed. This results in much smoother frame rate, assuming you can dynamically adjust the error metric to produce constant processing cost.

Do you mean keep a 2d frame with only stuff past a certain z (low error stuff) and re-project that into another frame buffer, then render close z 3d stuff normally into the same buffer, and display? Are you guys close to getting anything like this working? Would be curious to see it in action. Doesn't that mean you would have to render the low error stuff into a larger frame buffer with a wider field of view in case things shift during the re-project and blank edges get exposed? So it could mean you're always rendering a little more every frame to accommodate the re-project of the interpolated stuff. The other thing I don't quite get is at some point you have to 3d render refresh the 2d re-projected frame, wouldn't that cause a performance spike? Or maybe I've misunderstood what you meant...
 
Yes, the uneven cost of interpolated frames is a real problem. Games should always aim for stable frame rate (not best average frame rate). If you program your game to interpolate every other frame, your game will have lots of micro stutter (just like bad SLI/crossfire implementations). Delaying the cheap frame (vsync) helps a bit, but that pretty much spoils the idea, because you are just waiting for the vsync in the cheap frame (there's no gain by idling the GPU for most of the frame). Of course you can do tricks, and issue some processing for the next frame at the end of the cheap frame, but it's quite hard to load balance everything perfectly.

It's better to reproject every frame partially (the sections with low error metric) and refresh (render fully) only parts that are needed. This results in much smoother frame rate, assuming you can dynamically adjust the error metric to produce constant processing cost.

The Lucas Art's work shown in the star wars game do not display any micro-stutter (it does entire frame interpolation within predictable timings, for the price of motion-blur, making it completely free for games that operate at 30fps with motion blur, making them 60fps without the blur)

I can't believe I'm not agreeing with you (with my %0.01 knowledge compared to yours) but why would you have to make your GPU do nothing during the cheap frame? This is certainly not the case in the Star wars example, where the GPU starts working on the non-extrapolated frame one after another, and displays the cheap frame in the middle. (Only problem on Xbox360 was when you fall under 30fps for actual rendered frames, it required non-standard (or non-allowed?) way of framebuffer switching on the XBox360, as far as I remember, which would require some changes from MS, but there was no problem on the PS3)

http://www.eurogamer.net/articles/digitalfoundry-force-unleashed-60fps-tech-article?page=2

Tho I think render only what parts are needed approach would be good, like how I imagined a 1bit mask would do, wouldn't that make it more unpredictable, or wouldn't making it predictable be hard? (dynamically adjusting error metric to allow similar processing times between frames)
 
pretty interesting, the argument is this interpolation is cheaper than doing motion blur and look better, though it still doesn't change the gameplay is at 30fps.
 
but why would you have to make your GPU do nothing during the cheap frame?
Sebbbi mentioned in a previous post (which maybe you missed?) that load balancing is difficult; presumably, this star wars game (not sure which one you mean) uses art and level design tailored for the engine, so that it would result in (at least reasonably) predictable performance.
 
pretty interesting, the argument is this interpolation is cheaper than doing motion blur and look better, though it still doesn't change the gameplay is at 30fps.
Gameplay input and calculation is 60 fps (game engine decoupled from renderer). Visual look and movement cues are 60 fps. Only immediate visual response is 30 fps, and I doubt many gamers could tell an extra 1/60th second lag on their play. For games where you really need 60 fps feedback, you obviously need a 60 fps game so this isn't an option, but it's not being presented as a substitute for a perfect 60 fps game. For 30 fps games that'd look better at 60 fps with greater visual feedback, or 60 fps games that need the higher framerate but where the gameplay could be 30 fps and the picture look better, frame interpolation sounds a good idea.
 
I know many trained people (do the same thing over and over) can develop remarkable skills and sensitivities. I know a lady who had to count bank notes as part of her job. At first she'd count them out, but after a while she could riffle through a pile of notes and tell if there were fifty or not. I'm reasonably confident some majorly hardcore fighter nuts could tell the difference between a 30 fps and 60 fps input lag.

I suppose 'inhuman' isn't quite what we think. Skills beyond the average human is 'inhuman', and yet anyone can achieve such better-than-the-norm skill with practice.

It's also worth noting that sensitivity to timing isn't the same as immediate reactive response times. A flashing light > button press response time is of the order of tenths of a second, but a pattern that warns the brain of an upcoming event (flashing in time) can be sensed with some accuracy. A 30th second delay between a key press and a sound on a musical instrument renders it unplayable. A 60th of a second delay in the same is passable, but good musicians need lower.
 
A 30th second delay between a key press and a sound on a musical instrument renders it unplayable. A 60th of a second delay in the same is passable, but good musicians need lower.
Actually, as long as the lag was predictable your brain would adjust and compensate (would take a bit of practice, but soon you wouldn't notice the lag and would actually expect it to be there. :))

I wouldn't be surprised if oldtimer MIDI keyboards had on the order of ~1/30th of a second lag, considering the mechanics, glue logic and that the MIDI link ran at a whopping 115kbit/s, and that a keystroke required a few bytes to be transferred...but you'd never notice it anyway after a short while. :)
 
Actually, as long as the lag was predictable your brain would adjust and compensate (would take a bit of practice, but soon you wouldn't notice the lag and would actually expect it to be there. :))
Nope. There's a reason DAWs work on providing the lowest possible latency. You play a note in response to one just played, and hear the results. To achieve that with significant lag, you need to play irrespective of what you hear, which means decoupling your manual activities from your hearing of them, which is a decidedly unnatural condition. Maybe with practice you could accommodate lag, but you'd have to accept a ghastly experience for some time before that'd happen.

In the case of lag in games, for the average gamer, they can tell a delay between a button press and an action of however much time. It's certainly very noticeable in FIFA, for example, because the latency has to factor in long and short presses, so you can't react immediately with a button press. 1/30th of a second is probably small enough that no-one would notice (I think 2 or 3 frames is actually pretty unnoticeable in the average game), but from the response of hard-core fighter players, I think some of them would notice a frame lag much as musicians notice it when playing digital instruments.
 
Computer games are different because you need to react to unexpected situations, but I have yet to see the music instrument which comes at you with a sliding tackle out of the blue... ;)
 
It's certainly very noticeable in FIFA, for example, because the latency has to factor in long and short presses.

Just a small note, that the latency of "realistic" sports games is a special case, even when short and long presses were to not affect what your player did, you'd have to wait for the support foot to come down and kicking leg to strech out and then hit the ball which takes time, otherwise it would look very unrealistic for a player to change to shooting stance. Much like you'd expect the ball to bounce back into the hand before shooting in a basketball game.

To get back on topic, load balancing would not be too hard for such a game, no? If you can make sure the game runs stably at 30fps v-sysnced with motion blur, you can upconvert it to 60 fps. Many games are able to achieve this already. This technique obviously could have microstutter if you were to upconvert variable frame rates as you can't really guess how long a frame will take to render, I hope I'm not missing something here.
 
This might be interesting:

Start watching from 0:47. The original video has many usefull links on its description too.
 
The Lucas Art's work shown in the star wars game do not display any micro-stutter (it does entire frame interpolation within predictable timings, for the price of motion-blur, making it completely free for games that operate at 30fps with motion blur, making them 60fps without the blur)
The motion interpolation technology was not used in the final Force Unleashed 2 product. It was only a prototype that had some issues (links can be found in the game's Wikipedia page describing the issues). We also had a motion interpolation tech prototype running in Trials HD (2009), but we also removed it before the launch, because of some image quality issues (in some specific corner cases).
I can't believe I'm not agreeing with you (with my %0.01 knowledge compared to yours) but why would you have to make your GPU do nothing during the cheap frame? This is certainly not the case in the Star wars example, where the GPU starts working on the non-extrapolated frame one after another, and displays the cheap frame in the middle. (Only problem on Xbox360 was when you fall under 30fps for actual rendered frames, it required non-standard (or non-allowed?) way of framebuffer switching on the XBox360, as far as I remember, which would require some changes from MS, but there was no problem on the PS3)
It's not a simple as that. If you want to use vsync (to prevent tearing), you need to wait for the vertical refresh to swap the frame. If you start waiting too late, you have lost a refresh (wait lasts until the start of the next frame). Under normal circumstances (when not using low level hacks or multiple HW ring buffers of recent hardware), you cannot submit any draw calls when you are waiting for vsync (vsync blocks the CPU core, and only one core can submit the calls). As the GPU runs asynchronously, the CPU doesn't know how much GPU time the draw calls will take, and thus doesn't know exactly when to stop submitting draw calls and start waiting for the vsync. If you submit too many draw calls before the vsync, you will drop a frame (16.6ms lost time), if you submit too few, you will lose some GPU time (GPU just waits idling). That's the reality when you are using a standard graphics API. On PC you need to accept that, but on consoles, you could do some low level hacks to improve the situation. I don't know enough PS3 specifics to guess how it could be done with it, but I definitely have some ideas how you could hack around this limitation on x360 (using XPS and predicates). Of course you could improve the situation using adaptive vsync instead, but it just trades the issue with tearing: The more you guess wrong the draw call timings on CPU side, the more you will tear. It is impossible to exactly predict how much time the GPU will spend on a certain draw call (as exact prediction of any complex program sequence takes as much time as executing the sequence).

Of course you don't need to have perfect prediction to use majority of the interpolated (cheap) frame's GPU time to perform draw calls. You will only lose some of it. But that's not the only downside of motion vector interpolation. In order to have good quality, you need to interpolate between two frames (extrapolation will give you bad artifacts when objects bounce/collide). This kind of interpolation will add half of 30 fps frame's length of extra latency, as you can only show the interpolated frame after you have finished both frames. The real second frame will be shown half frame (one 60 fps frame later). So in the end, you will see slightly higher input lag than you would see in a normal 30 fps game.

In Trials HD we used an alternative way to add "extra" frame without running the complex shaders and lighting. We did render the geometry again on the "extra" frame, but we used a dirt cheap texture re-projection shader instead of the heavy material/lighting shaders. This way the "extra" frame had the most recent object position/rotation data from the physics engine (which was running at 60 fps). The frame latency was identical to a real 60 fps game. The downside of course compared to motion vector interpolation is the extra geometry pass cost (around 2 ms in our case), but that's still a huge gain compared to the 16.6 ms full frame cost. But that wasn't the case why the technique was ultimately dropped (as performance was almost doubled). We dropped it because of the graphics glitches caused by the reprojection. With these older DX9 era GPUs the performance wasn't good enough to refresh only the parts of the scene that were over a predermined error metric (or a dynamic one calculated by estimating the scene cost). A modern DX11 compute shader driven rendering pipeline can do that much more efficiently. I expect to see similar techniques becoming more popular in the future (if 1080p + 60fps becomes the preferred choice).
 
You can't simply write a value/s into the appropriate hardware register to make the GPU flip screen buffers while it's already running a display list?

Also, at least older video hardware could generate hardware interrupts at a certain beam position on the screen, which means you didn't have to spend an entire processor's worth busywaiting, checking for say, vblank period... Isn't this functionality present in this day and age?
 
I expect to see similar techniques becoming more popular in the future (if 1080p + 60fps becomes the preferred choice).

You made my day. Scratch that, you made my console generation :)

Thanks for the very detailed yet easy to grasp explanations, as always.
 
The motion interpolation technology was not used in the final Force Unleashed 2 product. It was only a prototype that had some issues (links can be found in the game's Wikipedia page describing the issues). We also had a motion interpolation tech prototype running in Trials HD (2009), but we also removed it before the launch, because of some image quality issues (in some specific corner cases).

It's not a simple as that. If you want to use vsync (to prevent tearing), you need to wait for the vertical refresh to swap the frame. If you start waiting too late, you have lost a refresh (wait lasts until the start of the next frame). Under normal circumstances (when not using low level hacks or multiple HW ring buffers of recent hardware), you cannot submit any draw calls when you are waiting for vsync (vsync blocks the CPU core, and only one core can submit the calls). As the GPU runs asynchronously, the CPU doesn't know how much GPU time the draw calls will take, and thus doesn't know exactly when to stop submitting draw calls and start waiting for the vsync. If you submit too many draw calls before the vsync, you will drop a frame (16.6ms lost time), if you submit too few, you will lose some GPU time (GPU just waits idling). That's the reality when you are using a standard graphics API. On PC you need to accept that, but on consoles, you could do some low level hacks to improve the situation. I don't know enough PS3 specifics to guess how it could be done with it, but I definitely have some ideas how you could hack around this limitation on x360 (using XPS and predicates). Of course you could improve the situation using adaptive vsync instead, but it just trades the issue with tearing: The more you guess wrong the draw call timings on CPU side, the more you will tear. It is impossible to exactly predict how much time the GPU will spend on a certain draw call (as exact prediction of any complex program sequence takes as much time as executing the sequence).

Of course you don't need to have perfect prediction to use majority of the interpolated (cheap) frame's GPU time to perform draw calls. You will only lose some of it. But that's not the only downside of motion vector interpolation. In order to have good quality, you need to interpolate between two frames (extrapolation will give you bad artifacts when objects bounce/collide). This kind of interpolation will add half of 30 fps frame's length of extra latency, as you can only show the interpolated frame after you have finished both frames. The real second frame will be shown half frame (one 60 fps frame later). So in the end, you will see slightly higher input lag than you would see in a normal 30 fps game.

In Trials HD we used an alternative way to add "extra" frame without running the complex shaders and lighting. We did render the geometry again on the "extra" frame, but we used a dirt cheap texture re-projection shader instead of the heavy material/lighting shaders. This way the "extra" frame had the most recent object position/rotation data from the physics engine (which was running at 60 fps). The frame latency was identical to a real 60 fps game. The downside of course compared to motion vector interpolation is the extra geometry pass cost (around 2 ms in our case), but that's still a huge gain compared to the 16.6 ms full frame cost. But that wasn't the case why the technique was ultimately dropped (as performance was almost doubled). We dropped it because of the graphics glitches caused by the reprojection. With these older DX9 era GPUs the performance wasn't good enough to refresh only the parts of the scene that were over a predermined error metric (or a dynamic one calculated by estimating the scene cost). A modern DX11 compute shader driven rendering pipeline can do that much more efficiently. I expect to see similar techniques becoming more popular in the future (if 1080p + 60fps becomes the preferred choice).
I wonder if a similar technology could be used to improve 3D stereoscopic capable games, and use the same technology interchangeably for framerate interpolation OR 3D rendering when the user wants to play in 3D.

I say this because this technology seems to be very similar to what you posted a few years ago on an article about 3D rendering:

"The reverse re-projection surface pixel cache system I used in early Trials HD development builds could really easily support stereoscopic 3D rendering," RedLynx's Sebastian Aaltonen posted on the Beyond 3D Forum.

"Stereoscopic views would need only very little extra processing power as the geometry passes are really cheap (they are basically just simple dual projected texture geometry with no light calculation, no shadow calculation etc). Stereoscopic rendering is really a poster child for all pixel reusing techniques."

http://www.eurogamer.net/articles/digitalfoundry-stereo-3d-article?page=2

I gotta admit I am much more interested in immersive 3D experiences than in framerate interpolation, but both technologies sound like they are related.
 
Back
Top