Generating realtime 3D facial animation from a single 2D picture

[one]

http://www.tv-tokyo.co.jp/wbs/2005/08/26/movie/tt.ram (Realvideo from a TV news program)
http://www.tv-tokyo.co.jp/wbs/2005/08/26/toretama/tt.html (TV Tokyo page for this segment of the TV program)

This technology is called 'Motion Portrait' and developed by Sony-Kihara Research Center that designed Graphics Synthesizer for PS2. It generates realtime (30fps) 3D facial animation from a single 2D picture, and is possible on an ordinary PC (you don't have to have a massive render farm). As you see in the movie, you can add facial animation to anything including pineapples and anime characters. The researcher says in the movie its application to games is interesting, for example animating your face or an anime character face in a game. I think with CELL it's almost inevitable to see it in action in PS3 games.

A Japanese page in Sony-Kihara Research Center briefly mentions this Motion Portrait technology (http://www.sony-krc.co.jp/research_epl2.html). At the bottom of this page it shows its history of developing 3D CG hardware engines.

1988 - Sony-Kihara Research Center was founded
1990 - Developed the world's fastest rendering engine (1.25 million poly/sec - 16 parallel processing)
1993 - Developed the graphics board for NEWS workstation (the world's first hardware Texture Mapping Engine for desktop computer)
1999 - Developed PlayStation 2 Graphics Synthesizer (the world's first Embedded DRAM Architecture, the world's fastest engine)
80 million PS2 shipped (at the end of 2004)
2003 - Developed Mobile Graphics Engine AGNES (ISSCC 2004) (the world's fastest per watt)

It seems an OpenGL/ES-compliant 3D CG IP core (for mobile devices and automobiles) is their recent interest.

 

[Titanio]

Very impressive.

To clarify it's one static 2D image? They don't just map video motion? If so that's very impressive, it looked very seamless. The rendering quality is also very high, but I guess that might just be a consequence of mapping something from real-life straight into 3D (no room for "bad art" ).

It'd be kind of scary to be able to map your face into a game with that kind of quality.

To they mention how much time it takes to get from the time the photo is taken to the time the 3D head is there and ready for use? Also, is this something that'd be ready for primetime soon, or more a long term thing?

 

[one]

To clarify it's one static 2D image?

Yes. Apparently it's important to take an original picture with blank face expression as the reporter woman does. Animation is currently composed of 16 facial expression templates mixed, but extensible with unlimited patterns according to the researcher.

To they mention how much time it takes to get from the time the photo is taken to the time the 3D head is there and ready for use? Also, is this something that'd be ready for primetime soon, or more a long term thing?

They say "instant" so not so long I guess. As they actually emphasize it's possible on home PC without special hardware, I assume it's ready now as a software.

 

[Shifty Geezer]

As all human faces are composed of the same however-many-muscles, from a 2D photo you can work out where those muscles would lie and adjust a basic template to fit, then apply all the different muscle animations to this transformed template.


Just watched it. Very impressive results. Has scarey possibilities in forging people being where their not though. Wouldn't take much to start faking videophone calls

 

[Titanio]

Yes. Apparently it's important to take an original picture with blank face expression as the reporter woman does. Animation is currently composed of 16 facial expression templates mixed, but extensible with unlimited patterns according to the researcher.

They say "instant" so not so long I guess. As they actually emphasize it's possible on home PC without special hardware, I assume it's ready now as a software.

Cheers, very interesting.

This could have interesting applications on the development side too. How long would it take for an artist to model a character of that quality? And how many could reach that kind of quality? Sourcing from actual people may be the way forward. It'd be very impressive if we had human character models of that quality next-gen..I mean the demo was a little reminiscint of the Alfred Molina demo from E3, but instead of having complex 3D scanners, just one photo? That's gotta be interesting to developers.

 

[MechanizedDeath]

Kojima has just found the way to extend those codec scenes in MGS4. :lol Seriously, you can't rotate the face past a certain degree, but for any video-phone-style action, this has to be pretty useful. I can see them rigging up super-realistic faces on the cheap for an MGS4 codec by just using a few pictures of real people. And it'll no-doubt look and work a lot better than anything they could try with 3D.

The guy behind this has some serious creative talent. His portfolio is mighty impressive. PEACE.

 

[Bad_Boy]

real media grr...
anybody have a different format of the video? i wanna see too

edit:
hmm... i got real player, but it still wont play. what are you guys using?

 

[EndR]

Wow! impressive!

I can see this when Sony introduces the whole "Minority Report"-Interface... I don´t know why but it sure sounds coool...

 

[MechanizedDeath]

real media grr...
anybody have a different format of the video? i wanna see too

edit:
hmm... i got real player, but it still wont play. what are you guys using?

Google Real Alternative, or grab it off download.com. I use it with the included media player classic. Matter of fact, I don't use WMP at all, just media player classic. All the codecs I can want works with it. And it's small and easy to install. PEACE.

 

[Bad_Boy]

Google Real Alternative, or grab it off download.com. I use it with the included media player classic. Matter of fact, I don't use WMP at all, just media player classic. All the codecs I can want works with it. And it's small and easy to install. PEACE.

thanks man, that worked perfect!


that video is amazingly impressive. i cant wait for this tech to be implemented into next gen

 

[BlueTsunami]

I can possibly see this being used for Tony Hawk and general sports games where the faced can be mapped onto something. Even though it states that it can only be used for scenarios like Metal Gear Solid's codec sequences (where you can only see the face as a 2D image). By the time the PS3 is out and this tech is able to develop, they can possibly get this going.

I also seem to remember one of the Tony Hawk games being able to use the Eye Toy to map your face and use it for your custom created character.

 

[Bad_Boy]

I could see it in fighting games, where you pick your character or character select screen, and your characters head/face is in the background in high detail moving around.

maybe also in scenes where a lot of objects are in the background like planes or ships to give the feeling of thousands of units in the background in high detail.

or just static objects that sit off to the side of the enviroment in racing games or sports games. like imagine if this was used in a colleseum where thousands of fans are sitting in the stands in high detail.

this could get even more impressive quick, i just hope devs use and implement it.

 

[MoeStooge]

Reminds me of a much, much better version of this:

http://veepers.budweiser.com/service/Start