Project Natal: MS Full Body 3D Motion Detection

[-tkf-]

http://www.eurogamer.net/videos/e3-project-natal-milo-demo?size=large

Milo demo, hints at a few things, notice how she puts on the googles. I like the touch with her reflection in the water.

http://www.eurogamer.net/videos/e3-project-natal-b-roll-footage

Notice the res on the shadows, they seem well defined, and the movements are picked up when she moves around, the video does cut quick and fast, most likely to hide in perfections.

 

[Crossbar]

The shutter isn't ... but that simply sits in front of the image sensor, it doesn't put a limit on resolution.

OK, Sorry I thought it was part of the sensor.

Anyway the sensor may not be bog standard anyway depending on the spec they are going for. The PS Eye has extra large pixels to make it work in low light conditions Natal may also have that.

 

[Crossbar]

Natal is not based on 3DV technology according to Alan Greenberg.

Doesn´t mean it could be very similar though.

 

[Arwin]

I'm updating the first post with bits of tech-talk I've come across that sound reliable and relevant. If you have anything to add or credible reason to dispute something, let me know in this thread and I will update the first post. My goal is to focus the discussion that way and keep the level up.

I also want to do the same for the other motion controls, so that if we have discussions comparing the three, we can base our discussions on some good technological background and keep the level up there as well.

 

[liolio]

Natal is not based on 3DV technology according to Alan Greenberg.

Doesn´t mean it could be very similar though.

Alan also hint they plan to milk the 360 till at least 2015
I start to really wonder about what Natal really includes in term of computational resources and memory. It's more and more clear to me that the system to run as intended need quiet some resources. I would not be surprised if some rumours are true and the thing could be costly.
I wonder how MS will market this, they will have a bunch of different SKU at hand. it's getting troublesome, imho Ms will have to clarify its offering while keeping one sku close to the 199$ price point.
I would like to see only two SKU, the xbox 360 and Natal 360 (better naming wanted!! ).
The xbox 360 being an "arcade 360" with enough GB of flash memory to support data caching and some downsloads. By 2010 16GB may be reasonable for a product in the 200 euros range.
The Natal 360 would be the same + obviously the natal system and be sold @ 299$ (Ms will lose money on the hardware).
HDD should be sold separetely in both cases.

The bothering part is the actual arcade and the fact that it doesn't support caching, going forward till 2015 it can be a problem. It's even more of a problem if we consider that future HDD less 360 could very well support this feature. Not too mention that the system doesn't not match in anyway MS view of future. I don't know how many arcade systems are in the wild but I think that MS should think forward and avoid the problem to expand by bumping the necessary amount of flash memory in the arcade sku soon.

 

[Neb]

The potential is huge to emotinally involve the player. Reaching out with your hand to help up a person that is loosing grip o na cliff edge/hole while the AI is having eye contact with player. Many possibilites to use it in gameplay and interactive cutscenes. Tech of the generation if executed well.

 

[obonicus]

Natal is not based on 3DV technology according to Alan Greenberg.

Doesn´t mean it could be very similar though.

Maybe that's where the $200 price tag comes from?

 

[pjbliverpool]

Most other 360 peripherals work with the PC as well. I wonder if MS will intergrate functionality for this device into Win7 (or add it in later).

The potential there is simply staggering. Having the capability to control your desktop without any peripherals. Imagine controlling media centre or web browsing through hand gestures and voice commands. Minority report here we come!

 

[patsu]

Maybe that's where the $200 price tag comes from?

3DV claimed sub $100.

 

[liolio]

VB: What does Natal mean?
SK: Natal is a city in Brazil, which is where one of our key development team members if from. Natal also means birth. We thought it was appropriate for what we are doing. This is the birth of the next-generation of home entertainment.

Time to spam some mail boxes!!! we want answers guys
Actually the whole venturebeat interview with Shane Kim is really worse the read

 

[obonicus]

3DV claimed sub $100.

Yes, but 3DV's not making this. Someone here said $200, someone else said that 3DV was $100. So we're back to, well, not knowing anything about the price.

 

[DrJay24]

I doubt MS knows the price yet, if they can get $100 for a $5 Wi-Fi device, I'm sure they will get at least that much for a state of the art imaging device.

 

[Shifty Geezer]

Eyetoy detect hands movement alright I thought.

Can Natal detect fingers movement ? You can do guitar heroes without the controller ? That would be cool, kinda like air guitar that people do.

Sony solution can utilised rings to detect fingers, assuming that bulb can be made small enough.

Any optical solution Sony can use, MS can use too. Unless Sony's implementation includes some other tech like a transmitter in the target device, they have no advantage.

This blog explains the tech implementation. There's a 'point cloud' of 3D values, one for each pixel of the sensor, measured as 'time of flight', so varying IR absorption by different materials won't impact the result. Will other ir sources? I imagine not. That 'point cloud' has to be interpreted as belonging to the person and which bits correspond to which limbs. The techniques sound fabulously effective, but this also means fine motions are going to be damned hard to detect. eg. Hold your hand in a ball and then rotate it a bit left and right. What the camera sees will be a cluster of points for the hand, but will it have enough detail and be able to track it well enough to have the user controlling a sword or pencil? Will props work okay? Broader motions seem the order of the day.

Edit : Just read via Eurogamer an MTV interview:

In an interview with Tony Hawk, who is at E3 promoting his new skateboard-centric peripheral, “Tony Hawk Ride,” we asked whether he was concerned about Project Natal. Microsoft’s experimental tech promises to rid the world of controllers, and even showed off a kid playing a skateboard game by just standing on his carpet. Yep, no expensive gear required! So does this mean Hawk’s game will be obsolete?

“We actually tried to get [Tony Hawk Ride] running with a camera, but it wasn’t accurate enough. Plus you can’t really do ollies, grabs and flips when you’re just standing on nothing.”
​

Although without saying for sure that it was the Natal system and not just a camera, it's inconclusive. But it does go with what I expect, that Natal is intuitive but lacks accuracy, and is better suited to EyeToy like applications. I can't see Natal as is managing Jedi Lightsabre Duels.

 

[MfA]

Will other ir sources? I imagine not.

Depends on how they do the calibration pulse and how much time there is between calibration and measurement. If they are captured by the same sensor in 2 separate passes strong enough IR sources which change in intensity during that time will screw things up (certainly possible since we are talking about ms).

With a shutter they could in theory use multiple image sensors to do calibration at the same time as measurement. If they use a special image sensor which "simply" allows you to discharge the capacitors in sub ns range times then you could do the same by using only say half of the pixels for calibration and half electronically shuttered pulses for measurement. With both these methods any light source defacto constant during the measurement would not affect the result.

It's probably easier to just generate a very strong IR pulse and shout all the other sources down though.

 

[fearsomepirate]

The potential is huge to emotinally involve the player. Reaching out with your hand to help up a person that is loosing grip o na cliff edge/hole while the AI is having eye contact with player. Many possibilites to use it in gameplay and interactive cutscenes. Tech of the generation if executed well.

All they've done is show a couple of half-broken or very weird tech demos, with no release date, price, or games, and people are envisioning a whole new radical world of gaming.

When and if this comes out, it's going to be the 360 version of the Eyetoy. It's not going to change the way you play Splinter Cell or anything like that.

Imagine controlling media centre or web browsing through hand gestures and voice commands.

People imagined controlling Link's sword with a physical swing, then discovered they got tired very, very quickly when it actually happened. Browsing through 40 gigs of music by waving your arms instead of clicking on a scroll bar will get very old as soon as the novelty wears off.

 

[Neb]

All they've done is show a couple of half-broken or very weird tech demos, with no release date, price, or games, and people are envisioning a whole new radical world of gaming.

They might sheed some more info soon.

When and if this comes out, it's going to be the 360 version of the Eyetoy. It's not going to change the way you play Splinter Cell or anything like that.

Lend me your magic 8 ball.

People imagined controlling Link's sword with a physical swing, then discovered they got tired very, very quickly when it actually happened. Browsing through 40 gigs of music by waving your arms instead of clicking on a scroll bar will get very old as soon as the novelty wears off.

Yet people buy Wii to play games with a stick. I'll rather use my hand without the need to hold a 150-200gr stick in my hands and waste my wrists and the risk off it flying into the Tv or out of the window. Might get sued and end on the headlines ""Dildo" jumps out of window and nearly kills a woman".

 

[liolio]

Edit : Just read via Eurogamer an MTV interview:
Although without saying for sure that it was the Natal system and not just a camera, it's inconclusive. But it does go with what I expect, that Natal is intuitive but lacks accuracy, and is better suited to EyeToy like applications. I can't see Natal as is managing Jedi Lightsabre Duels.

Hum... I question the relevance of Hawk's interview.
First he have to sell his stuff. Secondly how Natal could keep of what's going on?
Tracking the player body won't help much in most case aside of jump and push.
Track an hypothetical board is not an option, too fast too hard, board flying in the room
Basically Natal should have to track the kind of impulsion you're taking to guess that you're not only jumping but adding rotation/spinning.
As it states himself:

“We actually tried to get [Tony Hawk Ride] running with a camera, but it wasn’t accurate enough. Plus you can’t really do ollies, grabs and flips when you’re just standing on nothing.”

On top of that your legs are in line which might not help processing.
It's clearly not possible and it's not an indication of Natal lack precision whatever its precision will be in the end. And it's too be seen how much Hawk board is more gimmicky than anything.

 

[RobertR1]

Any optical solution Sony can use, MS can use too. Unless Sony's implementation includes some other tech like a transmitter in the target device, they have no advantage.

This blog explains the tech implementation. There's a 'point cloud' of 3D values, one for each pixel of the sensor, measured as 'time of flight', so varying IR absorption by different materials won't impact the result. Will other ir sources? I imagine not. That 'point cloud' has to be interpreted as belonging to the person and which bits correspond to which limbs. The techniques sound fabulously effective, but this also means fine motions are going to be damned hard to detect. eg. Hold your hand in a ball and then rotate it a bit left and right. What the camera sees will be a cluster of points for the hand, but will it have enough detail and be able to track it well enough to have the user controlling a sword or pencil? Will props work okay? Broader motions seem the order of the day.

Edit : Just read via Eurogamer an MTV interview:
Although without saying for sure that it was the Natal system and not just a camera, it's inconclusive. But it does go with what I expect, that Natal is intuitive but lacks accuracy, and is better suited to EyeToy like applications. I can't see Natal as is managing Jedi Lightsabre Duels.

How did they manage to try this with natal when the first dev kits for natal were only given out yesterday?

 

[Scott_Arm]

Check it out ... http://www.engadget.com/2009/06/03/project-natal-video-hands-on-impressions-and-further-details/

"The body tracking is truly impressive -- according to Kudo, it's picking up 48 joint points on the human body. As soon as we stepped into line in front of the box, the avatar immediately took on our stance and movements. And we mean really took them on -- little gestures with our arms, the posture we had, front and back movements -- it completely tracked them with accuracy. We did notice a bit of stutter during some finer movements, but overall the effect was impressive (and more than a little eerie)."

"The accuracy is far better than you would imagine it could be; it's very impressive stuff."


I think we should take the wait and see approach before making any assumptions about the accuracy of the skeletal tracking. I have no doubts that finger tracking is out of the question though. I'm also interested in knowing if it could pick up hand rotations like Shifty suggested.

 

[Asher]

Hold your horses! "Natal" does not mean a system where XB360 can understand human natural speech! The microphone array will enable voice extraction from ambient sounds (why aren't Sony using theirs?!?!), and some processing in the device may extract 'emotional stresses'. Understanding what the player is saying is down to the game code, or a new MS library, and this will be very limited.

How can we say how limited it is? Speech-to-text is fairly robust, especially for game use. MS Research has an entire group dedicated to this and they've been working on it for many years: http://research.microsoft.com/en-us/groups/srg/

While I'm sure it won't enable fluid conversations with AI entities, it'll definitely be strong enough to pick up key words and general gists of the speech to work into conversation-tree systems like in Oblivion or Mass Effect.

Of particular interest to the Milo demo are the following articles:
Understanding user's intent from speech: http://research.microsoft.com/en-us/projects/intentunderstanding/default.aspx
Microphone array processing and spatial sound: http://research.microsoft.com/en-us/projects/audioprocessing/default.aspx
Language modeling: http://research.microsoft.com/en-us/projects/language-modeling/default.aspx
Multimodal conversational user interface: http://research.microsoft.com/en-us/projects/slu/default.aspx
Speaker identification (who is speaking?): http://research.microsoft.com/en-us/projects/whisperid/default.aspx
SAPI (Speech API): http://research.microsoft.com/en-us/projects/sapi/default.aspx

MS' main advantage here is the enormous MS Research talent and workpool they have to draw from. This kind of stuff has been worked on in many different silos for years, and Natal seems to just be putting them all together for a practical application.