Realtime frame interpolation upscaling 30fps to 60fps

[kagemaru]

Concerning this quote:

According to Andreev, if the game drops below 30FPS the PS3 is still able to carry out the "flip" at the right point between the two frames. However, on Xbox 360, Microsoft's TCRs - the technical rules which dictate what you can and can't do with its hardware - insist that all calls to the graphics hardware go through their own APIs, and there isn't an equivalent system in place on DirectX.

Is there a chance MS can add a system on DX to address this issue of this technique catches on?

On the other hand I'm a bit worried by these trends. There's been a relatively steady increase in the image quality of realtime 3D graphics ever since the release of the 3dfx Voodoo - we started from 16-bit bilinear filtered relatively low-res to 32-bit with HDR, anisotropic filtering, higher resolutions and so on.
But first we started to see upscaled images on the consoles, and now this frame interpolation is another case that may hurt the final image quality. I'm a little worried about breaking this trend...

Maybe I'm missing something, but wouldn't it improve image quality if a 30fps game looked to be running at 60fps?

 

[Laa-Yosh]

That's not about image quality, it's a completely different case.
The individual frames will end up looking worse compared to the 30fps version, due to both the interpolation and the artifacts (which may or may not be eliminated as the tech gets refined).

One could argue though that the less motion there is between individual frames, the less noticeable the loss in quality would be, with almost nothing to see when standing still.
However any kind of flickering is still more obvious and disturbing, and for now this technique does produce quite some flickering artifacts (arms of the character suddenly disappearing etc). We will have to wait and see, at least until there's a retail game using this interpolation trick. MLAA is getting adopted in more and more PS3 games, so there is a chance to see this tech as well.

 

[kagemaru]

I get what you mean now and agree.

One thing I'm also interested in is the AA method they developed for TFU2. If it's comparable to MLAA but possible on the PS3, PC, and 360, then that is quite the achievement.

[DLAA (Directionally Localized Anti-Aliasing) - our custom anti-aliasing algorithm implemented in Star Wars : The Force Unleashed 2. In terms of picture quality it is comparable with MLAA, but due to perception based probabilistic nature it features high temporal stability and implemented on both the GPU (X360, PC) and PS3 SPUs ]

I know this is off-topic, but I wonder how the performance hit of DLAA compares to MSAA on the 360 or did they develop this AA method due to MSAA not agreeing with parts of their engine. Makes me wonder why Epic (with all their tech knowledge) hasn't developed their own AA method since MSAA doesn't work on UE3 as it should

 

[Laa-Yosh]

High temporal stability is a good way to say it doesn't flicker like MLAA supposedly does. That's another 'replacement' technique I have some trouble accepting, although - having no PS3 - I have yet to check it out in person.
As for Epic, I dunno, I believe their R&D efforts are already focused on different platforms - yesterday's iPad UE3 demonstration was one thing, and I guess they're working on UE4 in parallel, too.

 

[homerdog]

I know this is off-topic, but I wonder how the performance hit of DLAA compares to MSAA on the 360 or did they develop this AA method due to MSAA not agreeing with parts of their engine. Makes me wonder why Epic (with all their tech knowledge) hasn't developed their own AA method since MSAA doesn't work on UE3 as it should

I believe Laa-Yosh was hitting on this, but they may feel it's a waste of time to invent a new form of AA when they already have working MSAA on DX10 PC hardware, and the next round of consoles will be capable of this as well.

Wow that was way off topic.

 

[Kasersky]

That's not about image quality, it's a completely different case.
The individual frames will end up looking worse compared to the 30fps version, due to both the interpolation and the artifacts (which may or may not be eliminated as the tech gets refined).

i dont know its additive not destructive. the 30 base frames remain untouched, the tech simply inserts an additional interpolated frame in between each true frame.

 

[Laa-Yosh]

Yeah, but some of those odd inserted frames are the ones that might have elements that are gone from the next original frame, or should be gone after the previously rendered original frame. This is what may cause the flickering.
The arm on the far side of the character should be covered up by his body, and in the Z/prepass render it is - but since it was there in the uninterpolated frame, the image will remain there. The example videos have some of this, check them out again to see what I mean. And there may be a lot of other, similar examples - flashes of lightning, fast moving particle effects, muzzle flashes etc.

 

[MJP]

i dont know its additive not destructive. the 30 base frames remain untouched, the tech simply inserts an additional interpolated frame in between each true frame.

Yeah but the information that's "inserted" is essentially pulled out of thin air, and riddled with artifacts. Which means your eye will naturally combine the artifacts with the "real" information.

 

[Kasersky]

ya im aware of the artifacts, especially if you move the camera in conjunction with a shift in movement, my response was more because you mentioned that individual frames looked worse.

also sebbbi mentioned how physics engines typically run faster than the rendering framerate which could be used to interpolate frames as well.

 

[Neb]

Carmack should look into this to make RAGE 30fps->"60fps".

 

[Laa-Yosh]

Carmack cares far too much for image quality IMHO
And the control response isn't as good as with true 60fps anyway.

 

[Rangers]

Carmack should look into this to make RAGE 30fps->"60fps".

Huh? Rage is already 60 FPS. Do you mean doom?

To me, a graphics slut, I'm a whole lot more interested in the whole 15>30 FPS thing than 30>60

 

[Laa-Yosh]

He means that id should redo everything to run at 30 fps with more detailed graphics and use interpolation for the 60 fps.

 

[assurdum]

Yeah, but some of those odd inserted frames are the ones that might have elements that are gone from the next original frame, or should be gone after the previously rendered original frame. This is what may cause the flickering.
The arm on the far side of the character should be covered up by his body, and in the Z/prepass render it is - but since it was there in the uninterpolated frame, the image will remain there. The example videos have some of this, check them out again to see what I mean. And there may be a lot of other, similar examples - flashes of lightning, fast moving particle effects, muzzle flashes etc.

I think you are too much critic with this simple but interesting technic, from what I have understood how work... and we could refine more.. to me it's really a good alternative & it's very pleasant to see the programer overtake the limits of the console to find a new graphic approach... my question is: could be possible to use this method to have a 'fake' high buffer on the ps3, in theory obviously,

 

[Rolf N]

If it works purely from one frame, it's not actually interpolation. The "inter" in interpolation means a result between two (or more) samples is produced.

So this is [strike]prediction[/strike] extrapolation, not interpolation. In terms of latency that's actually better of course. I'm not sure I like the visual results though.

 

[Simon F]

So this is [strike]prediction[/strike] extrapolation, not interpolation. In terms of latency that's actually better of course. I'm not sure I like the visual results though.

Surely extrapolation implies having 2 or more images and predicting what comes after**?


** or earlier, but that's not very useful for video.

 

[RudeCurve]

Is this technology similar to what's used in Cyberlink's PowerDVD?

TrueTheater Motion: select this option to enable frame rate upsampling, from 24 fps up to 60 fps, to make panning scenes, particularly in action movies, play more smoothly. The Smoother option will double the frame rate of the original movie content, while Smoothest can improve the frame rate of DVD content to 60 frames/s or 72 frames/s, depending on the output frame rate of your display.

 

[Shifty Geezer]

If it works purely from one frame, it's not actually interpolation. The "inter" in interpolation means a result between two (or more) samples is produced.

So this is [strike]prediction[/strike] extrapolation, not interpolation. In terms of latency that's actually better of course. I'm not sure I like the visual results though.

It involves some lookahead though, using partial data from the next frame. So neither interpolation, as you say, nor extrapolation, as Simon says. I'd say call it Tweening and be done.

Is this technology similar to what's used in Cyberlink's PowerDVD?

It's the same principle, but unlike video upscaling that uses 2 frames and interpolates between them, this technique uses one rendered frame and some data from the next frame. It trades speed for quality, most notably in losing the character detail and only really tweening the background, whereas solutions that have more processing power to play with will tween every pixel.

 

[Gubbi]

John Carmack, at the recent QuakeCon, talked about supporting faster input response by rendering a buffer larger than the image on screen (ie. render borders outside the normal viewport). Simple rotation+translation of this image would effectively act as a prediction of the next frame.

Since you're spending fillrate rendering a larger image it is not without cost. But, if your game is getting 40frame/s consistenly (with tearing), it might be a win to render 30 larger frames per second and display these at 60 Hz with every other frame being a 'fast' frame.

You'd get lower input latency as well as smoother pans.

Cheers

 

[Rolf N]

Surely extrapolation implies having 2 or more images and predicting what comes after**?


** or earlier, but that's not very useful for video.

As far as I understood it they use the color buffer plus per-pixel motion vectors. Sure, in some ways that's a transformed representation of more than one frame, but it is/can be generated in a single frame snapshot.

Shifty, to be honest I didn't really get the part where they associate a post-rendering physics update with the "old" motion vectors and color data. Damn magicians! One step at a time please.