Project Natal: MS Full Body 3D Motion Detection

the conference showed some girl sitting on her couch, mimicking holding a steering wheel with her hands up, driving a race car in a game.

that's cool :cool:
That's Mario Kart with a Wii Wheel. Was she making motorboat sounds with her mouth to indicate acceleration?


to follow up

http://www.time.com/time/arts/article/0,8599,1902208,00.html

Kipman also showed me a version of Burnout that had been set up to work with Project Natal. Burnout is a serious game, not just a tech demo — it's a polished, fast-paced racing game with high-end graphics, and I happen to have played a lot of it. With Project Natal, instead of using a joystick, you steer by holding your hands up in the air like you're gripping a steering wheel. To hit the gas, you move your foot forward along the floor. To brake, you move it back. To trigger the turbo boost, you do a gear-shifting, fist-pumping movement with your right arm. Awesome.


It takes a few minutes to get the hang of it. You tend to oversteer, since you can't quite believe this thing is going to pick up your movements, so you exaggerate them. But soon you start to trust it, because it does actually work. I couldn't detect any significant latency. And there's definitely an extra edge to playing a game with nothing between you and the screen but your clenched, white-knuckled fists. I'm a hard-core gamer, so I'm not the person Project Natal is targeting. I love my controller as it is. But the appeal of Project Natal is real. You could compare it to the difference between regular movies and 3-D movies: it puts you in the action in a way that nothing else could.]
 
@_@ What ?!

Is there a link or official source for this ?

No I don´t think there is, but the fact they didn´t give even a rough estimate when it would be released and say that they are just about to ship it out to developers makes it hard to believe it will be released this year.
 
Driving like that is quite tiring for the arms if you are playing for an extended time, anyone driving MarioKart or using the motion control option in Motorstorm et al knows this. It may be fun for shorter periods, but I don´t expect it to revolutionize the driving genre.

Natals strength will be found in other games.
Perhaps. I posted it mostly to show that a) it is working with real games and b) that it's not a joke or cute. Evidently, it works.
 
I feel Natal isn't ready for primetime yet. It's not coming out until 2010.

I feel that Microsoft should've held off on this, perfected it, developed several killer apps, as well as other decent games, saving it for 2011 and for the next-gen Xbox. I think that would've made a bigger impact. However, with Sony probably about to reveal their motion-control scheme, I can understand Microsoft not wanting to be left behind.

Also, I'm certain that Molyneux's Milo demo looked massively more impressive than what the reality is going to be. It's the motion equivalent of pre-rendered graphics.

It surved its purpose at E3. It overshadows competition. It builds excitement and expectations that will be carried out to the next year. And next year they will feed it again with actual games.

It was a great marketing move
 
The reports from hands on, behind closed doors experiences with Milo I've read suggest at this point Milo is still being driven by a developer during interactions. To what degree? Only Lionhead knows. But there's a lot of smoke and mirrors with that right now.
 

But you're missing the point. Driving with your arms out holding a fake wheel is at about the same level of silly as holding out a fisher price wheel. Neither is a particularly good approximation of driving, and waving your arms around is not a 'natural' equivalent to turning a wheel, unless your steering wheel has a limit of about 180 degrees. Very good points were raised when Mario Kart came out how the Wii Wheel was, well, stupid. Only the most misguided, brave souls actually say good things about sixaxis driving (excluding Warhawk maybe, but I'm still not convinced). Why are we giving this the benefit of the doubt?

And I'm not even mentioning feedback which you certainly don't have with a camera -- even the wiimote has rumble, for what it's worth.
 
i would love to play Oblivion with natal, talking with NPCs to get/solve quests would be awesome.
Hold your horses! "Natal" does not mean a system where XB360 can understand human natural speech! The microphone array will enable voice extraction from ambient sounds (why aren't Sony using theirs?!?!), and some processing in the device may extract 'emotional stresses'. Understanding what the player is saying is down to the game code, or a new MS library, and this will be very limited. If Oblivion or Elder Scrolls 5 was to respond to player speech, it'd be key words. Perhaps a script of choices the player could read. They could also go with 'say something in the emotion you want' so you could make something up in an angry voice or a sycophantic voice and have an appropriate response. That might be quite cool.

Don't expect Milo type interactions with NPC's though. No-one has demonstrated anything close to that sort of AI sophistication, and the chances of Lionhead having something is remote, both because worldwide efforts haven't demonstrated anything like, and Molyneux has claimed incredible AI time and again without delivering.

Edit : Eurogamer has a hands-on
On motion tracking -
As I move, the camera moves with me. Shift your body left and the camera pans left; tilt your body forwards and it zooms in, and so on. "Normally you'd be using the right thumbstick for this," observes Molyneux. It's a bit disconcerting, but in a good way; looking around to change your viewpoint feels a lot more natural than pushing a stick.
This sounds superb. What I was hoping from PSEye. I hope games integrate this, so the game camera actually follows the natural motions of players.

On voice recognition -
I try another direct question. "Have you had a nice day, Milo?" He smiles and nods, so I go for something more complex. "Did you enjoy Microsoft's E3 conference?" He's non-committal. "Are you looking forward to the rumoured unveiling of the PSP tomorrow?" Milo nods. The truth is though, he doesn't really know what I just asked. He understood I was asking a question by listening to the tone of my voice, but he didn't understand the words. Knowing that doesn't make the whole thing much less unnerving, though.
So, nothing here but clever tricks, as we'd expect. The voice pitch recognition is clever, but they'll need to adapt to other languages with these techniques. I can see pitch detection being inaccurate, and not used much in Natal software. Also a fair bit of Mylo isn't unique to the Natal system, and could be achieved with a webcam - it's as much smart programming than new hardware.

The review of the motion tracking seems very good. The 3D imaging aspect has quite possibly solved the issues holding back the 2D cameras. It could besome awesome hardware. Now we need a price!
 
No I don´t think there is, but the fact they didn´t give even a rough estimate when it would be released and say that they are just about to ship it out to developers makes it hard to believe it will be released this year.

Ha ha, I see what you're saying. The target date is informally and subtly hinted at by the Natal project name: Christmas. Needless to say, they would hope to shoot for this year, rather than next. They could probably hit it by controlling the scope. Afterall, the hardware is there (may be limited quantity), and software can be patched.

first time I saw it, it reminded me of this

http://www.youtube.com/watch?v=aPENA1Bpm68

Nah... the Eyedentify one is closer.

For EyePet, there are more ways to interact with the pet other than talking. e.g., You could touch and pet the thing. Touching the boy for too long is not so politically correct (May be Heavy Rain 2 ?).

The player could teach it new tricks, and presumably create/draw stuff for the pet to play. The pet's reaction could be silly or stupid but you probably won't frown. If the boy's reaction is lower than his expected IQ level (Gah gah goo goo), then the player may think the boy is retarded. The technologies showcased in EyePet is the drawing recognizer (like the "Fish" Tank video) and the touching (a la EyeToy). Regardless of what the player is actually saying, the pet can still act "cute" and perform its tricks to appear interactive based on the volume, frequency and tone. The Milo experiment does not rely on 3D camera (Very little, just to know where to look). The natural language recognizer is the key input.

For a free form experience like Milo, the boy needs to listen, decipher the accent, understand and form his speech. If it's canned speeches, it's going to sound like Eliza and her voice mail. In something like Eyedentify, at least you can use the mission context and another game character to drive the conversation. So even if the player doesn't say anything, the game can move forward. There is a lot of fundamental R&D needed for the Milo (or even Eyedentify) experience. I won't expect to see the exact experience in the mall in any Christmas just yet. A seriously gimped one may be possible.


EDIT: Just saw Shifty's post. Yes, they need to gimp the boy's intelligence and intellectual level to make it work.
 
I come to think some more about the possibility of the system and finding ideas is not tough at all!
It's would be really easy to implement tiny kid games (think put the proper shape in the proper hole, 2d/3d puzzles, etc).
Sport/fitness games are a perfect match.
Educational games, I think that language teaching could be a great match. Voice and facial recognition should really help here. Facial recognition could help for pronunciation, you could have the model and your avatar mimicking your mouth movements.
Various virtual room games, think the room for "pillow fighting", dining room for girl, meeting room, why not theater room, etc bunch of possibilities here too.

If done properly it brings so much possibilities for casual games/activities, attraction I think more and more that Ms should brand this on its own.
 
Yes, casual and educational games would work. I don't think MS should focus on unrealistic experiments. The advantage is "no controller", not "sexiness". In a classroom settings, no controller means low maintenance and less damage. Even normal educational title would have value.

Voice and facial recognition for young kids (depending on how young) may be difficult. Just focus on touch. There is still a lot of untouched potential in EyeToy.
 
Do you think that this may also help to down the development costs as it could be used to do mo-cap for cheap?
 
Development cost depends on many factors. You can probably do a cheap or expensive project using the same technology.

I don't see Project Natal as a technology to reduce game dev cost in general. When you do mo-cap, you get tons of 3D geometry data. When you do 3D camera, you really only get slices of 2D images. That's why the recognition can be fast.
 
Hold your horses! "Natal" does not mean a system where XB360 can understand human natural speech!

im aware of that

The microphone array will enable voice extraction from ambient sounds (why aren't Sony using theirs?!?!), and some processing in the device may extract 'emotional stresses'. Understanding what the player is saying is down to the game code, or a new MS library, and this will be very limited. If Oblivion or Elder Scrolls 5 was to respond to player speech, it'd be key words. Perhaps a script of choices the player could read. They could also go with 'say something in the emotion you want' so you could make something up in an angry voice or a sycophantic voice and have an appropriate response. That might be quite cool.

yeah thats what i want to see in real game, dunno if that may actually work well.
Also
They could also go with 'say something in the emotion you want' so you could make something up in an angry voice or a sycophantic voice and have an appropriate response.
could be very cool feature when u need to scare someone to get some informations about quest, objective or whatever.
 
Molyneux actually discusses the libraries used a little in the Eurogamer interview - Lionhead themselves went hunting through Microsoft's different libraries from all the divisions. For instance, the speech recognition component comes from Windows 7.
 
Yes, it would be fun to sneak up on a pack of Kobolds, shout 'Boo' and see them scarper. But that's not needing new hardware. This could have been done years ago. Just no-one thought to add it. Had one of these consoles launched with a camera by default, we'd probably have a more intersting game scene than we have now. And more gimmicks. Although, given the lacklustre application of sixiaxis motion control, I dare say developers wouldn't actually do a good job of innovating. Even on Wii, it's Nintendo leading the way mostly and the rest awkwardly following up the rear. That is an issue for Natal - it'll only be as good as MS's efforts. 3rd party won't make anything of it other than facsimilies of MS's efforts.
 
Molyneux actually discusses the libraries used a little in the Eurogamer interview - Lionhead themselves went hunting through Microsoft's different libraries from all the divisions. For instance, the speech recognition component comes from Windows 7.
So that aspect in games could be extended to any PC with a webcam of headset. What Natal brings that's unique is the 3D camera. The other bits could have been done with PSEye, and the fact it hasn't just goes to show what a waste some technology is and how dependant we are on out-of-the-box software developments.
 
Molyneux actually discusses the libraries used a little in the Eurogamer interview - Lionhead themselves went hunting through Microsoft's different libraries from all the divisions. For instance, the speech recognition component comes from Windows 7.

I'd imagine so, but the state of the art speech recognition won't be able to do what Milo did in the concept video. The automated help desk software works because their context focuses on a limited vocabulary.

I believe the best in the industry is from IBM (could be wrong here). It could transcribe a doctor's verbal diagnosis in near real time (lot's of technical terms helps).

But that's not needing new hardware. This could have been done years ago. Just no-one thought to add it.

When EyeToy was introduced, people did talk about using it to recognize emotion, via voice or/and facial recognition. I believe Phil Harrison mentioned it in an old interview (circa 2003, 5-6 years ago). I guess the devs are all tied up with regular game dev challanges. Now that they are getting better with basic dev, may be they can revisit it.

But you hit it right. The other major aspect is marketing. Wii works because Nintendo thought through the subtle issues and solved the key challenges. Then they make it fun just by looking at it.... and cheap enough.
 
Back
Top