Using Photographs to Enhance Videos of a Static Scene

Interesting new tech developed at Washington Uni, basically using high res still to enhance or edit video scenes. Watch the flash movie.

We present a framework for automatically enhancing videos of a static scene using a few photographs of the same scene. For example, our system can transfer photographic qualities such as high resolution, high dynamic range and better lighting from the photographs to the video. Additionally, the user can quickly modify the video by editing only a few still images of the scene. Finally, our system allows a user to remove unwanted objects and camera shake from the video.
1513129
 
So, in a nutshell, this is a technique where the video stream provides the depth info to the hi-res stills or applied post-effects?
But nonetheless, it's impressive to see it in action. ;)
 
Well I'm not sure, because there isn't really any depth info in a video, so they must somehow model the scene. They did talk a little about wrapping the photos into 3D space. The highres stills (and it's amazing they only use a few) are kind of used like a high-res mip map for selected parts of the video. The superfusion blending across the temporal stuff was impressive. I really liked the bit at the end where they removed the lamp post and sign from the flower stand - totally seamless.

We've reached that point where you can't believe anything you can see on screen, because that editing of a scene from a couple of photos and a couple of frame changes was incredible.

Even the the other applications (relighting the dark table scene, HDR'ing the library window, steady camming the shot of the guy on the roof) were pretty incredible. If this sort of thing can be put into the postprocessor of a video camera it would bring a huge advance in IQ and editing ability. Heck, even available for a modern PC would be great for home post production and editing work - it would be a great showcase for a GPU's processing power.
 
Well I'm not sure, because there isn't really any depth info in a video, so they must somehow model the scene.
There isn't in fact, but the temporal nature of the video stream could be used to construct an accurate enough scene depth info, where the magic is.

Now, there's a real incentive for all those CCTVs to get larger sensors. :D
 
Back
Top