Project Natal: MS Full Body 3D Motion Detection

Arwin

Now Officially a Top 10 Poster
Moderator
Legend
Project Natal was announced on Microsoft's E3 press conference. The technology uses a 3D camera and the SDK for it seems to support face recognition, voice recognition and fully body motion scanning in real-time.

Discuss the technology and its potential applications here.

Things we know about the technology:

Eurogamer interview:

Eurogamer: How does Project Natal work with Burnout?
Alex Kipman: Essentially we do a 3D body scan of you. We graph 48 joints in your body and then those 48 joints are tracked in real-time, at 30 frames per second. So several for your head, shoulders, elbows, hands, feet...
Say I'm tracking a wrist, which is what I do for Burnout. I can look at that on a single frame and I can see what direction, acceleration and confidence I have for that joint. Why is that interesting? Because it allows me to not only know where you are, but to know where you're going to be. This is how we do the directing and the predictive behaviour.
If you think about swinging a baseball bat, by the time you're halfway done with the swing, I know not only where you're going to end but when you're going to end. There are very precise and predictable ways so you can have that immediate payoff of my baseball bat hitting the baseball.
- http://www.eurogamer.net/articles/e3-post-natal-discussion-interview

The 3D sensor itself is a pretty incredible piece of equipment providing detailed 3D information about the environment similar to very expensive laser range finding systems but at a tiny fraction of the cost. Depth cameras provide you with a point cloud of the surface of objects that is fairly insensitive to various lighting conditions allowing you to do things that are simply impossible with a normal camera.
- http://www.3dvsystems.com/technology/tech.html

WRT precision - it's a time-of-flight camera, which means they emit a IR light ray and measure the time for it to return; they probably scan the room with the ray, and they can control it at will; one of the materials said they scan the entire scene in 5 frames; if they want to detect only hands, for example, they can focus mostly on the area around where the hands were last frame, but receive increased precision in return.
- http://forum.beyond3d.com/showpost.php?p=1299174&postcount=146

My own general summary based on what I know so far:

The camera can scan its full field of view in 5 frames, generating a point cloud similar to how for instance environments can be digitally scanned by laser these days. It uses this to find the body. Then it focusses on the body and generates a new point cloud which instead of wasting a lot of time and points on the whole field of view, now shows mostly the body. This information is mapped to a predefined skeletal framework that has 48 joints, to determine the body posture of the player in 3d. From here on, it will keep focussing on the body solely and compare each new 'frame' / point-cloud with the previous position, to determine several parameters for the joints, like position, speed and direction of movement, and also how accurate the software in the camera doing the analys thinks this information is ('confidence' per Eurogamer's own interview). It gives back this information at 30 frames per second apparently (also per the Eurogamer interview).
 
With regards to the Milo/Mylo? Demo that Pete did, I was thinking that perhaps theres a really strong possibility on an interactive education type component here. I mean if the rumours of the cost of the camera are true ($200), its one of the few applications which would entice people to pay THAT much for a peripheral.
 
I think that the milo type technology could revolutionize RPGs. Could you imagine interacting with NPCs in a RPG like you can Milo. Even the simple things like being able to navigate menus minority report style to find a netflix to watch will rock hard. I really hope the full body reconition works well. I would love to see Fight Night 5 with full body control. The possibilities are endless. I am really impressed with Natal.
 
According to Molyneux, Milo is what's been known for many years as the super secret Dimitri project at Lionhead. Quite interesting.
 
Yeah, after Fable 2 you can easily see why Pete likes the technology, and I think he's a good one to have around for figuring out cool things to do with it. And he has Lair as a nice example of what pitfalls to avoid too. ;)
 
Milo didnt realy impress me at all as a showcase for the new hardware. It has been done before to some extent with seaman and most of it would be possible with a standard webcam and mic
 
Looks to good to be true and my experience tells me when something looks to good to be true it usually is. It will be interesting to go back and compare the E3 presentation to what actually will done in games with this device.

If it is supposed to work with finger movement detection and detection of facial expressions on a bunch of people simultaneously spread over the room, it needs to have a crazy high resolution and the built-in computer power needs to be pretty massive to deal with it without adding to much delay.

No release date was given, not even a hint as far as I have seen. They must be working like maniacs to get the manfacturing costs down on this one.
 
There's the rub. The potential is enormous. The precendent is weak. We've seen tech-demos come and go for all sorts of optical devices. We've seen demos of "In the Movies" and "Lips" that looked far more effective than the final products. However, this is a new tech from an independent group rather than an MS creation, so I'm thinking the tech may well be a success. The big issue is implementation and cost. Something this good should be in other high-tech industries, not just games. Why isn't in in effect in remote-control applications?

So, my concerns are cost, effectiveness, and software. I haven't seen the vids yet. We saw EyePet last year, and that's taking it's time to appear. Some PSEye stuff has just not happened for years. When will the roll-out of Natal be, and will it be supported? Strangely though, I feel optimistic about this one. I dare say it's time MS got a Trump Card and won the limelight for a spell. I also imagine Sony are pretty gobsmacked (assuming cost isn't crazy!).
 
What's the precision I wonder ... for say finger movement to be able to select a menu item you need cm level precision.
 
It looked to me like everything shown in the trailer with the family in front of the tv was cg concept videos not actual games being developed. I think only the demo with the balls flying at you, the art thing, and milo were actual games using the hardware. The art demo was the only really impresssing one to me because of how closely the on-screen dummys actions were to the guys. Thing is the avatar demo looked terrible using the same tech.
 
Finger detection is highly unlikely. Vicon optical mocaps have a hard time with it using special reflective markers and a hige set of cameras - so I doubt it could work on such a small scale.
 
Having seen the vids on Gametrailers now, I dare say there's a bit of overpromising here. The family thing looked totally concept-rendered, as it were. I wouldn't be surprised if the family were actually acting out the content on the screen, rather than the screen content follwing their motions. the Milo thing showed some nice touches, but Molyneux's talk had me umming and nodding my head. His promises are always way beyond results. So a character that hears the emotion in the players voice and responds accordingly? More like it notices some vocal stresses and picks one of a few responses, but can get readily confused. I was thinkng just the other day how voice recognition was a mobile phone gimmick that never took off, principly because it was utterly unreliable. End Wars is the best voice-recognition we've got so far.

Also, though I don't think the Milo experience was staged, I'm scratching my head over how the camera can detect the hand positions so close to the screen shen she's fishing. It'd need a flippin' wide-angle lense to take that in, and then everything at any distance would be too small to interpret.

Milo offers some very immersive experience, but it's a set-piece, clearly. There are a finite number of voice recordings they could use (unless they have this as speech synthesis..!) and we all know how useless computers are at trying to simulate conversation. There is no way MS/Lionhead have cracked it, and got the XB360 recognising people's emotions and changing it's behave in accurate response. The fact Molyneux claims they have is almost absolute proof to the contrary :p

Add in the showing is behind closed doors, that means it's early days and they haven't got a smooth experience to present to the world. Will they actually be able to get one, or are they showing something too early that won't work?
 
Lionheads Milo project (using Natal)

video

pretty cool

oh and I'm pretty sure it was stated by Mattrick that Natal would be bundled with future consoles,

evidently I'm not the only one who heard that
 
Heh heh, it certainly looks like a concept video.

This is like Eyedentify in the PS3 launch year. I much prefer the blonde and brunette.


Tap In said:
oh and I'm pretty sure it was stated by Mattrick that Natal would be bundled with future consoles,

evidently I'm not the only one who heard that

If he doesn't specify the SKU and timeframe, then we may have to wait a while for that future (probably starting with the more expensive models, and depending on the field response).
 
Finger detection is highly unlikely. Vicon optical mocaps have a hard time with it using special reflective markers and a hige set of cameras - so I doubt it could work on such a small scale.
Vicon isn't using time of flight cameras, they can only deduce depth whereas a time of flight camera can measure it directly. Also Vicon systems generally have to record natural motion, not restricted motion which is ensured to not generate occlusion.

If Microsoft really can mass produce time of flight cameras cheaply it changes the ballgame completely. Vicon could pretty much scrap it's own markerless system for instance, so they are probably hoping Microsoft can't do it.
 
If he doesn't specify the SKU and timeframe, then we may have to wait a while for that future (probably starting with the more expensive models, and depending on the field response).


If MS does what it appears to me they want to do with it (based on the demonstration and commitment) I think they will want to give this away.

It may make them a wii contender and along with the most robust online service and 1v100 casual gaming etc., a possible front-runner for the most consumer bucks for several Christmas's to come. :smile:
 
oh and I'm pretty sure it was stated by Mattrick that Natal would be bundled with future consoles,

evidently I'm not the only one who heard that

That will not happen for a very long time, or else they would need to raise the price.

I would guess it will be released fall 2010, after real games have been shown at next E3.
 
Back
Top