grandmaster
Veteran
That's an interesting idea but how would you get lower latency? Assuming 60fps, let's say you move left... the image you'd be getting would be 16ms "old", surely?
Realtime frame interpolation upscaling 30fps to 60fps
- Thread starter Kasersky
- Start date
That's an interesting idea but how would you get lower latency? Assuming 60fps, let's say you move left... the image you'd be getting would be 16ms "old", surely?
Yes, but it would be a rotated/translated view of the 16ms old image.
Think of it this way. You move your thumbstick left, instead of waiting 33ms for the aiming recticle to move, you only wait 16ms. Entities wouldn't be updated (at all!), but you get more tactile feedback for movement.
I think Carmack was considering it to boost high frames per second to very high frames per second, ie 60Hz -> 120Hz, or even 100Hz->200Hz. The signal infrastructure isn't there at the moment to support these kinds of refresh rates.
Anyone who ever played QuakeWorld on a 200Hz server (with a CRT refreshing at 200Hz ) will tell you that the added fluidity makes for more precise aiming/shooting.
Cheers
For what it's worth, I've done quite a bit of experimenting with such a feature on my TV (a Panasonic G20). I'd always considered it a completely throwaway feature, but it actually works surprisingly well in many games. Third person games and hack'n'slashers make the input lag much less noticable than FPSes, and Darksiders (for one) saw a huge benefit from it.
I was very sceptical of such features until I saw it in action for myself. I never really noticed any artifacting either; the only case where it was noticable was in games that use a static reticule while the gameworld spins around.
I was very sceptical of such features until I saw it in action for myself. I never really noticed any artifacting either; the only case where it was noticable was in games that use a static reticule while the gameworld spins around.
Billy Idol
Legend
Could such an interpolation tech in theory be used for other game related stuff?
For instance for the game physics? When cloth physics are so difficult, half the update rate and interpolate it to the goal rate?!
Or for instance for real time shadow updates. I remember a Crysis 2 demo, where the shadow of a falling tree only updates with half (or so) rate - could this be interpolated as well to make things smoother?!
For instance for the game physics? When cloth physics are so difficult, half the update rate and interpolate it to the goal rate?!
Or for instance for real time shadow updates. I remember a Crysis 2 demo, where the shadow of a falling tree only updates with half (or so) rate - could this be interpolated as well to make things smoother?!
The TV frame interpolation is different though. It works only on finished color frames.
The tech in the OP uses intermediate color and non-color results produced in the game engine to compute an advance frame. It has a much more accurate view of which objects are supposed to move where and how fast.
Yeah, I've gone through the whole presentation now and of course you're right. It's pretty cool stuff. The healing filter is the most impressive part of it IMO.
It's been fairly standard practice for awhile in some genres to run all game-logic updates at half (or some lower number) of display FPS with interpolation. How far that goes depends on how latency sensitive particular game design is.Billy Idol said:
Update rates often vary on per-subsystem basis as well.
Refreshment
Regular
It is a shame this hasn't resurface in anyway or form and so little disscussion was made in here
http://www.youtube.com/watch?annota...0q&src_vid=w1sjRD7NSec&v=93GwwNLEBFg#t=15m53s
He calls it "time warping" and mades no mention of Dmitry Andreev research that was made public in 2010. I didn't know if making a topic about it, hopefuly the disscussion could be revived somewhat in here. To this day im amazed non of the graphics vendors or console manufacturers invested good money researchiing this since it would be quite a competitive advantage.
And speaking of the devil:
http://www.youtube.com/watch?annota...0q&src_vid=w1sjRD7NSec&v=93GwwNLEBFg#t=15m53s
He calls it "time warping" and mades no mention of Dmitry Andreev research that was made public in 2010. I didn't know if making a topic about it, hopefuly the disscussion could be revived somewhat in here. To this day im amazed non of the graphics vendors or console manufacturers invested good money researchiing this since it would be quite a competitive advantage.
Time warping is more about lowering the latency of head/mouse motion feedback, which is mostly rotation ... not about image based rendering of interpolated/extrapolated frames. He does talk about the latter, but separately.
Time warping simply works.
Interpolation/extrapolation might very well never work well for games. The problem with interpolation is that you actually have to introduce extra latency, you need two frames to interpolate in between, the problem with extrapolation is that you still get jerkiness when the extrapolation gets it wrong. Image based rendering with the game logic (and thus physics) running at the higher framerate might work ... but the extra work limits the gains.
Time warping simply works.
Interpolation/extrapolation might very well never work well for games. The problem with interpolation is that you actually have to introduce extra latency, you need two frames to interpolate in between, the problem with extrapolation is that you still get jerkiness when the extrapolation gets it wrong. Image based rendering with the game logic (and thus physics) running at the higher framerate might work ... but the extra work limits the gains.
Though I am not a big believer on the tech, I have to point out that you are incorrect there. The interesting bit about Andreev's proposition, is it is not interpolating between frames, but actually guessing the next one base on its velocity buffer. At worst case, latency is the same as if the algo wasn't used at all. Of course this makes the not fully correct assumption that velocities are constant, and any movement discontinuity will be incorectly represented. Aside from that there are all the artifacts that come with working in screen-space, and the dodgy accuracy of velocity buffers comonly used in game engines (though doing it more acurately and paying the prince in memory and performance is still likely cheaper than re-rendering a full frame).
I actually think that despite the theoretical incorrectness of all this, it wouldn't introduce terribly noticible artifacts, which is what really matters in real time rendering, but I think there are smarter ways to use the concept, like for post-antialiasing (crytek did it), better motion blurring, or reusing of static information from past frames for smart spacial upscalling ( Dust 514 did something along those lines, but without the warping). The full robust solution would be object-based selective time-stepping rendering of everything: Render different objects seperately at different framerates (anything from 15fps to 300) depending on their on-screen movement speed. Nothing trivial to implement, and if ever gets done, will be many years, and at least a couple hardware generations from now, but that would be a very non-redundant way to go about rendering.
So something which is not interpolation not having the issue proves I am wrong about saying interpolation having the issue?
That's some strange logic there.If you re-read my post you'll see I already made a distinction between interpolation and extrapolation ...
Time does really fly... I can't believe this is a 3 years old thread
Since Trials HD, we have done some research, but I don't think we will release anything using this kind of technology before we are ready to release a game with fully compute driven pipeline. This kind of stuff is hard to do efficiently by traditional rendering pipeline.
Straightforward frame interpolation techniques offer quite a big boost to frame rate, but at the same time create huge amount of small problems in image quality. If you cannot measure local error metrics and react to those (in a fine grained way), you will need to compromise a lot (render much more than required to hide the most glaring errors). But as we can see from the best commercial video compression codecs (Blu-ray for example), it is quite possible to reduce a lot of information (= processing cost & bandwidth requirement if we do the same for real time games) and still keep most of the image quality intact. As resolutions and frame rates grow, it's easier and easier to compress the data (and processing cost). Basically you just have to place your sampling points correctly spatially and temporarily and combine / reuse results cleverly. Fortunately computer graphics is not pure 2d image post processing. We have knowledge about our scene structure, depth, object boundaries and movement. TV interpolation techniques do not have this information, and thus must rely much more on (educated) guesses.
Since Trials HD, we have done some research, but I don't think we will release anything using this kind of technology before we are ready to release a game with fully compute driven pipeline. This kind of stuff is hard to do efficiently by traditional rendering pipeline.
Straightforward frame interpolation techniques offer quite a big boost to frame rate, but at the same time create huge amount of small problems in image quality. If you cannot measure local error metrics and react to those (in a fine grained way), you will need to compromise a lot (render much more than required to hide the most glaring errors). But as we can see from the best commercial video compression codecs (Blu-ray for example), it is quite possible to reduce a lot of information (= processing cost & bandwidth requirement if we do the same for real time games) and still keep most of the image quality intact. As resolutions and frame rates grow, it's easier and easier to compress the data (and processing cost). Basically you just have to place your sampling points correctly spatially and temporarily and combine / reuse results cleverly. Fortunately computer graphics is not pure 2d image post processing. We have knowledge about our scene structure, depth, object boundaries and movement. TV interpolation techniques do not have this information, and thus must rely much more on (educated) guesses.
Have you guys ever considered a hybrid approach like where you split the scene in two, near z stuff and far z stuff, and render the far z stuff at a much lower frame rate and interpolate that to 60fps, but render the near z stuff at full 60fps? I figure the near stuff is what changes position the most per frame so keep that at full 60fps rendering to avoid nasty interpolation artifacts, and try 10fps to 30fps interpolated for the far stuff and blend the final result together. Maybe pick the frame rate of the far stuff depending on camera movement, so keep it 30fps->60fps interpolated when the user is panning the camera around, but try 10fps->60fps interpolated when the user is mostly moving forward, or something in between based on how much the angle of vision has changed from the last frame.
How would you manage workloads on unpredictable player behaviours? That is, if you're rendering exactly enough to achieve 60 fps near-field and 30fps interpolated at distance, and then the player's movement changes and you can render the distant field at 10 fps, what do you do with the processing time you've freed up? You can't up the quality of the near-field stuff without it showing when you change distance rendering refresh rate. Seems to me all you could do is save some processing cycles but be unable to commit them to anything else as you couldn't depend on their availability - I don't see any value in developing such a system.
At this point it's more to see if such a system can be developed to where the typical user does not notice any artifacts. A proof of concept basically. Right now every frame rate interpolation that has been attempted is riddled with artifacts that most anyone can see, so as a first step see if you can make a setup where the artifacts are relegated to far z and see if anyone notices. If no one notices then good, get more aggressive with it, maybe always keep the backgrounds at 30fps regardless and see if anyone notices, rinse and repeat. The idea is start small and try to get something that is visually acceptable then build off that. Right now there isn't any interpolation setup that I know of which yields acceptable results, and personally I'm skeptical that near z stuff will ever be interpolation friendly.
With the game engine information available, I can't see why it wouldn't work. Any artefacts only need be present for 1/60th second in between true frames, which means the artefacts would have to be pretty horrendous to get noticed. I suppose worst case would be some aspects that don't appear to refresh as quickly because the interpolation is plain broken. Things like DOF or fast objects revealing scenery (eg. a car passing a crowd, you couldn't interpolate the crowd revealed in between frames) could prove nigh impossible to interpolate.
It's the sort of thing that'll probably get a breakthrough in academia before anyone starts putting it into a game, simply because the cost of toying around with it is likely hard to justify from a business POV.
Edit: Actually, the arcades could be a place to develop this. Coupled with foveated rendering, you could get some 10c the performance from a graphics card. One GTX 780 could render visuals to the user equivalent to 10x as much power by rendering much of the screen at lower quality and with lower framerates interpolated - photorealism for a lot of games would be on the cards.
It's the sort of thing that'll probably get a breakthrough in academia before anyone starts putting it into a game, simply because the cost of toying around with it is likely hard to justify from a business POV.
Edit: Actually, the arcades could be a place to develop this. Coupled with foveated rendering, you could get some 10c the performance from a graphics card. One GTX 780 could render visuals to the user equivalent to 10x as much power by rendering much of the screen at lower quality and with lower framerates interpolated - photorealism for a lot of games would be on the cards.
After listening to his talk, I have to agree with Carmack here, in that these techniques will become important once we get high level penetration for VR and much higher resolutions and framerates become necessary to avoid nausea.
Based on my experiences (granted, long ago) with a VR helmet, 4K and 120Hz should become a must have for that tech, and then it'll be more economical and practical to use high quality rendering for something like 60fps and then interpolate to 120. It'll also be much easier to hide artifacts and do the prediction and such, and on the other hand the need to up the frame rate by +100% will also become far more important with VR. Within the next 5 to 10 years we could get the computing power required for that, too.
But getting 60Hz for 1080p console games on a 1080p HDTV with this approach may still be too much of a compromise.
You can also see why Carmack joined Oculus, there's at least a decade's worth of R&D to look forward to, and this technology can easily become the next big thing. Augmented Reality is pretty exciting too, but that will probably have different, not graphics related kinds of issues and he's probably not as interested in those.
Being able to stream HD video on a pam sized device practically anywhere was SF even when the iPhone was introduced. IT is advancing at a breakneck pace, people always underestimate it.
Energy tech is at the other hand, it's always overestimated and yet we're still running on fossil fuels and practical electric cars are still beyond the horizon. We'll have AR well before those vehicles or fusion power.
Energy tech is at the other hand, it's always overestimated and yet we're still running on fossil fuels and practical electric cars are still beyond the horizon. We'll have AR well before those vehicles or fusion power.