archangelmorph
Veteran
Out of curiousity how do you expect them to mocap an eyeball??
You can't just stick little white balls on it like you can everything else..
*spin-off* Motion Capture and Animation in Games
- Thread starter Shifty Geezer
- Start date
Out of curiousity how do you expect them to mocap an eyeball??
You can't just stick little white balls on it like you can everything else..
Out of curiousity how do you expect them to mocap an eyeball??
You can't just stick little white balls on it like you can everything else..
Yes, you can.
RenegadeRocks
Legend
Well, as for me, the best part in these images are the eyes. Even in HS and KZ2 vids, even after the melodramatic acting the eyes felt dead. Even in photography, students are asked to focus on the eyes.If the eyes come sharp, the image feels strong and pleasing(portraits). In paintings too, we are supposed to keep the eyes sharp and defined to make it believable. Eyes identify a person, even in news videos, eyes are blurred or blacked out to hide the identity of a person. I think Quantic Dream realises that, and have made the eyes very detailed.Very reflective as in real life, feel wet and has the slight specular whre the eye ball touches the eyelid on the lower lid( omitted by most people in CG).The tech demo video showing the mocap of just eyes shows that they are working in the right direction.
While it is possible, the preferred method is to put a few markers on the eyelids and calculate the rotation of the eyeball from how it pushes the lids around. The cornea creates a bulge on the front and it isn't a perfect sphere anyway, so it can be done.
However, mocap is never good enough to be used as it is. The process goes like this, for body movement:
raw marker data -> filters and noise reduction -> retargeting to the character's skeleton -> another noise reduction and cleanup pass -> manual keyframe animation on top of mocap
Depending on what the director wants, sometimes the entire recorded motion is abandoned and replaced by keyframe animation. But some level of adjustment is always required, from basic stuff like planting feet on the ground or correcting proportional differences between the mocap actor and the character.
Facial animation is an entirely different case. Some movies do try full 'performance' capture, like Beowulf; but even there, a lot of manual animation is neccesary. In other cases, like with Gollum, Kong or Davy Jones, all the facial animation is keyframed (ie. manual).
Also, there are two approaches with facial animation and mocap; you can try to do a 1:1 mapping between markers and bones/clusters/etc. on the CG face. Or, you can try to analyze the data and detect facial expressions, and then drive manually created morph targets with the result.
What I'm trying to say is that eyes are usually relatively easy and quick to keyframe, also because most mocap performances are recorded on video for reference. So there's no real need to spend time with fixing eye rotation data.
Maybe I'm not entirely clear on the concept of keyframing, but as I understood, Heavy Rain was trying to sample eye motion to a much higher degree (so as to capture even the small eye motions, to better emphasize the 'acting'). Wouldn't keyframing get in the way of that, unless you use a whole lot of keyframes?
Still, compared to mocapping body movement, tracking eye motion seems like a much simpler task.
Most animation packages offer the ability to use layers in the animation and mix and blend between them, copy/paste keyframes etc. So you don't have to commit to either.
Keyframing basically means that for certain frames of the animation, you set the value of parameters like translation, rotation, amount of any given blendshape, light intensity etc. Then you can set the type of interpolation to use, from stepping and linear through various spline based stuff with additional controls.
Mocap on the other hand is usually 1:1 on frames/keyframes, so a LOT of data; but you can optimize it both to reduce the size of the dataset and to replace jittering/noise with smooth interpolated curves (which can, of course, sometimes remove the very fine details as well).
(In 2D animation keyframes were the most important parts of the movement, drawn by lead animators; and the rest of the crew had to fill the frames between them.)
The trouble with facial capture in general is that the movement is tiny, compared to full body motion. Even the markers are only 3-5mm in diameter, compared to the 20-30mm used on the body.
So the basic setting for a large active space means that these movements are much closer to the sensitivity limit of the system. Typical Vicom cameras have a resolution of 2 Mpixel, and have to cover 5-10 meter wide spaces. Even if you focus the lens on a single person, you have less then 1mm per pixel. So you get very, very messy data with a normal setup.
Sony uses a LOT of cameras to compensate in their 'performance capture', 100+ instead of the more common 16-24. This is VERY expensive stuff; and it's well known within the industry that even Beowulf had to rely on the animators for a lot of the work.
People have also experimented with head mounted face cameras, but they're not stable enough and can't really record 3D data from a single 2D image.*
The other option is to record the facial animation separately, with the person sitting in front of 8-12 cameras arranged in an arc. This gives very good accuracy and each marker can be tracked by 4-5 cameras, resulting in more precision. You can also stabilize the markers to the head, so the facial features will stay put
But then you won't get synchronization with the existing body movement, and even the eyes can't be precise, as there's no other actor or enviroment to interact with etc. So this one would also require keyframe animation for corrections, although it's obviously faster then starting from scratch.
This is why most productions simply abandon not just the capture of the eyes, but the whole face completely. They'll get better full body data and the extra money for the facial stuff is better spent on more animators.
*It's worth to note that ILM has some super secret stuff used from POTC2 onward, that has allowed them to capture body movement from the movie camera's view, without any extra cameras. Noone has any idea how this is possible, but even this requires considerable tweaking, of course.
Keyframing basically means that for certain frames of the animation, you set the value of parameters like translation, rotation, amount of any given blendshape, light intensity etc. Then you can set the type of interpolation to use, from stepping and linear through various spline based stuff with additional controls.
Mocap on the other hand is usually 1:1 on frames/keyframes, so a LOT of data; but you can optimize it both to reduce the size of the dataset and to replace jittering/noise with smooth interpolated curves (which can, of course, sometimes remove the very fine details as well).
(In 2D animation keyframes were the most important parts of the movement, drawn by lead animators; and the rest of the crew had to fill the frames between them.)
The trouble with facial capture in general is that the movement is tiny, compared to full body motion. Even the markers are only 3-5mm in diameter, compared to the 20-30mm used on the body.
So the basic setting for a large active space means that these movements are much closer to the sensitivity limit of the system. Typical Vicom cameras have a resolution of 2 Mpixel, and have to cover 5-10 meter wide spaces. Even if you focus the lens on a single person, you have less then 1mm per pixel. So you get very, very messy data with a normal setup.
Sony uses a LOT of cameras to compensate in their 'performance capture', 100+ instead of the more common 16-24. This is VERY expensive stuff; and it's well known within the industry that even Beowulf had to rely on the animators for a lot of the work.
People have also experimented with head mounted face cameras, but they're not stable enough and can't really record 3D data from a single 2D image.*
The other option is to record the facial animation separately, with the person sitting in front of 8-12 cameras arranged in an arc. This gives very good accuracy and each marker can be tracked by 4-5 cameras, resulting in more precision. You can also stabilize the markers to the head, so the facial features will stay put
But then you won't get synchronization with the existing body movement, and even the eyes can't be precise, as there's no other actor or enviroment to interact with etc. So this one would also require keyframe animation for corrections, although it's obviously faster then starting from scratch.
This is why most productions simply abandon not just the capture of the eyes, but the whole face completely. They'll get better full body data and the extra money for the facial stuff is better spent on more animators.
*It's worth to note that ILM has some super secret stuff used from POTC2 onward, that has allowed them to capture body movement from the movie camera's view, without any extra cameras. Noone has any idea how this is possible, but even this requires considerable tweaking, of course.
I have an idea for a totally different mo-cap technology that'd, in theory, work with actors on set in full costume, but it's dependent on a tech I only have the loosest understanding of, so there may be technical limitations. I am surprised that mo-cap is both incredibly clever in doing optical point tracking, while also such a low-tech solution, following daft-looking pingpong balls! Has it really been the only pracitcal solution people have invented? Or have ILM found an alternative (my idea?) but looking for something a little more sophisticated?
Those markers are actually pretty expensive, IR-reflective stuff. The cameras only see in infrared too, as far as I remember. And there is quite some software complexity behind it...
There's an old method using electromagnetic fields, but it requires proper enviroment (reduce distortion - even large pieces of metal can mess it up - and cables attached to the sensors. It can now use radiowaves to communicate so the actor doesn't have to be plugged in, but the number of sensors is fixed and thus the system isn't as flexible as Vicon's optical stuff. You can use that to track equipment, props, armor etc. as well as the above mentioned face stuff.
It's kinda like asking why we only have internal combustion engines. Works well enough so far, considering all circumstances.
There's an old method using electromagnetic fields, but it requires proper enviroment (reduce distortion - even large pieces of metal can mess it up - and cables attached to the sensors. It can now use radiowaves to communicate so the actor doesn't have to be plugged in, but the number of sensors is fixed and thus the system isn't as flexible as Vicon's optical stuff. You can use that to track equipment, props, armor etc. as well as the above mentioned face stuff.
It's kinda like asking why we only have internal combustion engines. Works well enough so far, considering all circumstances.
Thanks, Laa-Yosh, for that interesting background material. I have seen a few jitters in Heavenly Sword, but I always thought they were glitches in the graphics engine more than anything else. This is definitely not something I would have realised myself.
This small feature about Killzone 2's cutscenes is a nice short and quick example of what some of the highest level of mo-cap is, which indeed, like Heavenly Sword, includes facial expressions, though you can't see the capturing of eye movement in this youtube video.
Actually here's the Heavenly Sword bit, that includes the eyelids.
And this is interesting also - a user movie showing a tour of some of Sony's facilities on display, including 3d object and face scanner and facial motion capture (price of the latter machine included - hey, only 6 million ).
http://www.gametrailers.com/player/usermovies/83926.html
This small feature about Killzone 2's cutscenes is a nice short and quick example of what some of the highest level of mo-cap is, which indeed, like Heavenly Sword, includes facial expressions, though you can't see the capturing of eye movement in this youtube video.
Actually here's the Heavenly Sword bit, that includes the eyelids.
And this is interesting also - a user movie showing a tour of some of Sony's facilities on display, including 3d object and face scanner and facial motion capture (price of the latter machine included - hey, only 6 million
).http://www.gametrailers.com/player/usermovies/83926.html
Last edited by a moderator:
I'm no animator but I'd say that you need to use the entire face to differentiate.
Blank stare with eyebrows slightly raised, jaw hanging a bit with the mouth still open, eyes themselves slightly more open then usual.
Sharp glare with head tilted forward so you see a bit of the whites on the lower side of the eyeball, and the upper part of the iris is covered by the eyelids. Eyebrows slightly furrowed, lips pressed together. Maybe scale the pupils down a bit too.
Depending on the art style, it can be cartoonishly overdone or pretty subtle. And yes, these are acting decisions, so either the mocap performer or the animator has to make them. Animators are basicaly actors using someone else's body for their work, with the ability to go back and finetune it through as many iterations as neccessary.
We also tend to cheat reflections on the eyes now, there are many ways to do this. Special reflection maps, compositing tricks (blur, adjust levels, contrast, intensity etc).
About running passes on mocap... there can be many reasons.
- The data is noisy, or some markers were lost and the movement is missing completely for some frames. This usually requires skill to properly reconstruct the missing segment with the right dynamics and of course it should not stand out from the rest. We regularly got data where obvious noise either wasn't removed, or it was smoothed out to a level that it became lifeless. We even have dedicated folders in every project to store these usually hillarious videos...
- Retargeting. Giant ogre/monkey/robot doesn't look like your performer, so you have to compensate the posture, the width of shoulders or whatever. But even simple human characters will almost always require adjustments. Again, unskilled tech guys can do crazy stuff here.
- Changes requested by the director. Timing, poses, dynamics, whatever. Doesn't like the actor's take at all, likes parts of this and that take, and so on.
- Stuff that's never been there in the mocap. Fingers, toes, hand/foot contact with objects, floors, other characters. It's just not possible to do well, and animates quickly with inverse kinematics, pose libraries - and seasoned animators
Eyes can belong to this case too.
The GG guy is kinda exaggerating there IMHO
but it's still a good movie to show the process. There's two prior stages, a short treatment and a complete script but those aren't really visual stuff so no need to talk about them.And I don't recall LOTR doing actual facial capture, certainly not for Gollum. I've been part of an online community where we've been exchanging a lot of info with the Weta guys responsible and he was all keyframed, with hand crafted blendshapes; it took four guys several months to build the shape library for the fellow, after years of R&D! But it had an amazing level of control...
The GG guy is kinda exaggerating there IMHO
And I don't recall LOTR doing actual facial capture, certainly not for Gollum. I've been part of an online community where we've been exchanging a lot of info with the Weta guys responsible and he was all keyframed, with hand crafted blendshapes; it took four guys several months to build the shape library for the fellow, after years of R&D! But it had an amazing level of control...
I was under the impression Gollum was motion captured facially - if not all power to the animators to get Andy's expressions spot on. The result of facial motion capture in HS was pretty spectacular in certain scenes...many of Andy Serkis's performances were uncanny...and you wouldn't think expressing such emotion and even humour would be possible with a videogame character. Same goes for Flying Fox and Whiptail...an excellent moment when she shows a surprised and angry expression on her face. Unbelievable.
So unless you've played the game and seen it first hand it's a bit hard to go into too much detail. It's still the standard for motion captured characters, at least facially. I expect Heavy Rain to surpass it and Killzone 2 to equal it.
Out of curiousity how do you expect them to mocap an eyeball??
You can't just stick little white balls on it like you can everything else..
Contact lenses with dots painted on them
No, the DVD bonuses went into great detail
D), and this is why Andy Serkis was deemed unsuitable to be nominated for an Oscar. They captured his performance, but all the animation was hand-done using Andy's performance as a reference. Thus the actual thing we saw was the product of computer animators, not exactly Andy, and so the Oscar for best actor couldn't, in the judges' eye, be awarded to him.
I very much doubt Killzone 2 to equal it, simply because that requires a level of acting that I very much doubt they'll have. The actors in Heavenly Sword did some great things with their faces, and had a much better context for this also. I don't think Killzone's warzone will host anything near as much quality.
As a Dutchman by the way, it's still a bit uncanny in and of itself to see a basically Dutch team create such a high profile PS3 title. The games that have come from here have been ... well ... really rather horrible. (Though at least someone from the Netherlands worked for the GTR team I think)
There's a localisation company here though that does a lot of work I think for a lot of (Sony) games, taking care of a lot of the voice acting, subtitles, translations, spelling and so on. If I were young and badly payed and lived a little closer to the company, I'd have loved to work there!
Of course Killzone is now a big and international team, as predictably they couldn't get all their qualified personnel from the Netherlands alone.
Still, these guys impress me so far!I very much doubt Killzone 2 to equal it, simply because that requires a level of acting that I very much doubt they'll have. The actors in Heavenly Sword did some great things with their faces, and had a much better context for this also. I don't think Killzone's warzone will host anything near as much quality.
As a Dutchman by the way, it's still a bit uncanny in and of itself to see a basically Dutch team create such a high profile PS3 title. The games that have come from here have been ... well ... really rather horrible. (Though at least someone from the Netherlands worked for the GTR team I think)
There's a localisation company here though that does a lot of work I think for a lot of (Sony) games, taking care of a lot of the voice acting, subtitles, translations, spelling and so on. If I were young and badly payed and lived a little closer to the company, I'd have loved to work there!
Of course Killzone is now a big and international team, as predictably they couldn't get all their qualified personnel from the Netherlands alone.
Well of course the acting will be very different, not as theatrical. But their budget is huge...so they can employ very good actors (perhaps not Andy Serkis and Stephen Berkoff standard) but still, i was more talking technically.
No, the DVD bonuses went into great detail
Ooh ok. Well that must have taken forever.
How long is a typical session from capture to cleaned up ?
Similar threads
