Improving character animation in games

[Shifty Geezer]

LA Noire has demonstrated incredible realism in their direct motion capture footage, but as Laa-Yosh will readily point out, it's a very limited technology in application, akin to Dungeon Quest's linear video footage offering the best 2D visuals for a decade but the worst gameplay.

Well, Remedy are after the animation crown, and are claiming they've surpassed Rockstar with their latest tech. This article says they capture human models at half-millimetre accuracy, and with 64 facial captures, can blend the full range of human expressions. The reporter says the results are impressive.

This isn't a new approach, blending key models, so what is it Remedy are doing differently? Is it just the volume of capture, or does the solution rely in a novel blending mechanic? Given developments like Euphoria's behaviour simluation, is there a prospect of truly natural characters in terms of motion and facial expressions, or will control and interaction issues limit what can ultimately be attained in games? eg. Uncharted still suffers from captured poses not matching the scenery Drake stands on. Euphoria could animate him realistically, but the cost would be human-style limits on motion, which would be perceived as unresponsive controls by players. Like Lair, implied control rather than direct control probably won't go down well. So going forwards, is there even much point in developing new animation technologies in most games, or will they remain the preserve of niche titles?

 

[Laa-Yosh]

At first glance I'd say it's the PR that they're raising to new levels.

Some behind the scenes info.
FACS stands for Facial Action Coding System, which has been developed in the sixties by psychologists to create a standard for describing human facial expressions. The individual movements are based on neurology and a lot of research and tweaking. Using five levels of intensity, a set of 50-60 Action Units can define nearly all possible facial expressions. The results will of course look different on people with different facial structure, anatomy, build and such, but it's as close to a universal descriptive language as possible.

During the making of the Lord of the Rings movies, Weta has utilized FACS to define Gollum's facial expressions (and copy them as much as they could from Andy Serkis, the actor) and IMHO the results were quite successful. In the past decade FACS has become a standard in VFX, it is of course somewhat built upon and modified and such though. Avatar, Benjamin Button, probably Davy Jones as well, everyone uses it as the start.


The first game to utilize FACS was - as far as I know - Half Life 2; which isn't such a surprise once you learn that Bay Raitt, the guy responsible for Gollum, went to work at Valve after the LOTR movies and has been there ever since (Team Fortress 2 and Left for Dead have cool facial animation too).


So, what Remedy is doing is probably related to this approach. Scanning every base FACS expression from the actor and using it as reference for animated normal maps, even animated color maps, is well known stuff in movie CG - but it won't magically solve every problem.
The main trouble here is that AUs almost never appear on their own, but rather in combination with other AUs - compressing the face into an angry expression may involve a dozen of them or even more. But when the skin and flesh starts to push or stretch against itself, the looks of the individual AUs no longer apply, and the mixture will look very wrong. The Final Fantasy CG movie was a victim to this, they had no solution and thus the animators simply avoided to push the expressions too far, creating wooden animation.
Gollum (and most CG creatures nowadays) solve this by using additional "fixer" or "corrective" poses, hand sculpted by the artists to fix every combination used. This meant thousands(!) of such shapes for the old hobbit because of how big a range he had to cover with his facial expressions.
But I don't think that Remedy will go this far, it's far too work and memory intensive, so it's probable that more complex expressions will look just as wrong on their characters as well, and lip sync too.

LA Noire overcomes this problem by only digitizing and storing the shapes that are actually used during the animation, and streaming them from disc during the playback. They may have hundreds of thousands of frames of animation altogether but they only keep the minimum in memory at any given time, and they can create them automatically using their scanning equipment and software.

In the end, it's hard to compare the two approaches technically, because you can't literally animate using LA Noire's tech, it's all pre-recorded and baked and then played back without any additional input. I haven't seen what the faces do during normal gameplay where characters should try to react to interactive events immediately but I'd guess the quality is not the same.
What Remedy is talking about on the other hand is a fully interactive animation rig, they can even drive it procedurally during gameplay (display various emotions, do eye tracking, breathing, whatever). I'd also guess that the actual quality will be a mix - storing less data means more detailed textures and shaders, but less realistic deformation.


Also, it might be interesting that the guy Remedy has hired was up until recently a member of Image Movers Digital, the studio responsible for all of the Bob Zemeckis mocap movies and has transitioned from standard mocap to facial mocap during the last two movies, Christmas Carol and Mars Needs Moms. He certainly has the technical knowledge, but it's also worth remembering that neither of these movies were that well received by audiences.

 

[MrSpiggott]

For action games anyway, I'm not sure we'll be able to improved movement animation a great deal beyond what we have now without ruining playability . We all like quick changes in direction, but people can't physically turn at such speed. I'm not sure I'd like to play an Uncharted game where you'd have to worry about momentum or which leg you were pushing off of when you wanted to turn or jump.

 

[Shifty Geezer]

Most informative, thanks! Sounds like what's needed is a new mathematical model for interpolating key poses without needing custom blends for every combination...

 

[Laa-Yosh]

No, interpolation isn't the answer, for example certain combinations produce different wrinkle and fold patterns compared to the sum of the base AUs. Lip shapes are some of the most complex stuff, too.


As for general animation of the body, it's another interesting topic but I'm not too well informed about that one.

 

[Shifty Geezer]

I wasn't thinking a straight arithmetic blend.

 

[Arwin]

It is interesting you bring that up, because I had just been showing my wife some just released footage of The Witcher 2, which looks really nice, but the first thing she commented on negatively was the animation of the lead character. It is definitely an immersion breaker, for me also.

Personally I am quite happy with Uncharted 2 already. The base system is really good. It just needs some tweaks for better physics here and there, although you may need a character modeled more after a young Jackie Chan in order to make some of the moves look more physically convincing while keeping them smooth and responsive. (wouldn't that be cool?)

Really, when I look at this again, while not playing myself, it strikes me even more how good this game looks. Somewhere in this video there's a moment where he's running while reloading his shotgun. Those animation blends are really great.

This video also reminds me that this game has so many chapters and environments, its just stunning.

 

[patsu]

I like the character animation in KZ2+3, and the facial in U2 and LA Noire. The Last Guardian catdog animation looks interesting too.

 

[nightshade]

What I like about Uncharted is that the game has a right balance of everything, if they make the animations too realistic then the game starts to suffer from control issues (GTA4), now I don't need to tell you which out of these two games is the game with smoother and better controls. The animation looks realistic, blends realistically but at the same time the speed is just right so as to make it feel responsive.

 

[patsu]

... and Chapter 20 in U2. It's the one that made me cast a vote for U2 (against KZ2).

Character animation in Uncharted was calculated and blended on the fly from a library of existing animations. They also added custom logic to prevent malformed body parts (e.g., prevent chest or arms from flattening when twisted -- using muscle emulation).

KZ focused more on integrating animation tightly with AI (so every little AI decision can be reflected on-the-fly via animation ?).

I'm hoping U3 can combine both.

 

[Billy Idol]

FACS stands for Facial Action Coding System, which has been developed in the sixties by psychologists to create a standard for describing human facial expressions. The individual movements are based on neurology and a lot of research and tweaking. Using five levels of intensity, a set of 50-60 Action Units can define nearly all possible facial expressions. The results will of course look different on people with different facial structure, anatomy, build and such, but it's as close to a universal descriptive language as possible.

During the making of the Lord of the Rings movies, Weta has utilized FACS to define Gollum's facial expressions (and copy them as much as they could from Andy Serkis, the actor) and IMHO the results were quite successful. In the past decade FACS has become a standard in VFX, it is of course somewhat built upon and modified and such though. Avatar, Benjamin Button, probably Davy Jones as well, everyone uses it as the start.

Thanks for the info!

I saw a presentation of those Avatar guys about this FACS this year in Reno at a math conference(!).

Was super impressive how the face animations can be broken down in some basis and then recombined resulting in all sorts of different expressions (if I remember correctly, it takes much more 'effort' to look angry, than to look happy...or the other way around?!).

With respect to animation, what destracts me the most (beside the facial animation of course and the often poor lip syncing) is that the transition between different animations is often poorly executed and looks often weird...for instance when you input new comandos on you joypad, cancel a movement or something like this...

 

[Shifty Geezer]

With respect to animation, what destracts me the most (beside the facial animation of course and the often poor lip syncing) is that the transition between different animations is often poorly executed and looks often weird...for instance when you input new comandos on you joypad, cancel a movement or something like this...

This is the problem with marrying fluid, natural animations, with unnatural, responsive control. If you are running a character one way, then want to run backwards, and then run forwards again, in a typical shooter say, physically a human can't do this. To animate realistically would require lots of input lag as the avatar physically adjusts against momentum. At the beginning of this generation things like Euphoria looked really promising, but it appears realistic character animation is never going to happen in action games. Could work well in something like ICO.

 

[Sigfried1977]

I'm actually very happy with the state facial animation is at right now. Body movement on the other hand, now that's an area that could stand a ton of improvement (also another vote for Uncharted 2. Not entirely realistic, but incredibly cohesive and without any glaring weaknesses)