Kinect technology thread

[patsu]

Right. Probably too slow for controlling i.e. the aim of a player character then. But how about, say, in a squad based shooter where you'd want to issue commands like: "Team two: Flank left to checkpoint A, then hold until GO. On GO, breach and proceed to checkpoint B". I can think about several immersive ways of doing this with either/or/a combination of voice, gesture, and pointing controls (all of which should, to some degree, be available with Kinect); except for the standard controller getting in the way.

It all depends on the game. In MAG, it's possible for enemies to sneak into your area while you're still coordinating among the team. If you add voice and gesture to the picture, it'd just complicate the whole interaction (More chances of screwing up). The user needs to memorize all the supported gestures and voice commands for each game (The semantics may vary).

Even for Move people, I suspect most will get tired of elaborate motion control. That's probably why Anton talked about adding "macros" to the controller to replay recorded moves.

Kinect, Move, Wiimote+, Vitality Sensor, WiiFit, etc, are all about the games. I don't think developers want to do elaborate controls for technology sake.

 

[dragonelite]

Think that 360 players would find this kind of stuff that trackerIR does way more interesting then aiming with kinect. Maybe Rare can implement some of these features in a hopefully planned perfect dark sequel. But not sure if kinect camera is fast enough or precise enough for it.

http://www.youtube.com/watch?v=9wXx3vMy_AQ

 

[Arwin]

It all depends on the game. In MAG, it's possible for enemies to sneak into your area while you're still coordinating among the team. If you add voice and gesture to the picture, it'd just complicate the whole interaction (More chances of screwing up). The user needs to memorize all the supported gestures and voice commands for each game (The semantics may vary).

Even for Move people, I suspect most will get tired of elaborate motion control. That's probably why Anton talked about adding "macros" to the controller to replay recorded moves.

MAG though is multi-player only, so voice recognition is pointless imho - you talk to real people so you don't need the game to know what you are saying, unless perhaps you want it to automatically map those commands into graphical feedback like waypoints and routes. But then I think you'd be better off with advanced strategy interface in the manner that I think you and Shifty discussed earlier (and which I agree would be really, really cool - combine RUSE with MAG, so to speak).

Being able to control your arm to make gestures about what to do when voice chat is not available because the enemy would overhear you could be really cool also, but not easy to do with either controller as neither does fingers really well. Would love to see developers try anyway though - even a menu with predefined gestures would be neat. I'm all for anything that makes online gaming more strategic and less run-and-gun but I may be the ony one.

Kinect, Move, Wiimote+, Vitality Sensor, WiiFit, etc, are all about the games. I don't think developers want to do elaborate controls for technology sake.

Should want anyway.

 

[AzBat]

Think that 360 players would find this kind of stuff that trackerIR does way more interesting then aiming with kinect. Maybe Rare can implement some of these features in a hopefully planned perfect dark sequel. But not sure if kinect camera is fast enough or precise enough for it.

http://www.youtube.com/watch?v=9wXx3vMy_AQ

It's something that I could see them do. Just not sure they would go with needing the TrackClip Pro attachment. I guess they could sell specially designed headsets that have the TrackClip Pro integrated in.

BTW, here's the specs of TrackerIR5...

• Raw Sensor Resolution: 640 x 480
• Reporting Resolution: 96,000 x 72,000
• Field of View: 51.7°
• Resolution/Horizontal Degree: 1850 subpixels/degree
• Sample Rate: 120 fps
• Response Time: 9 ms

Kinect's CCDs are at 640x480 too and they claim a 57° FOV. Just not sure about the sample rate & response time. But it looks like the Kinect hardware is probably technically capable of doing everything TrackIR5 can do. The rest is all up to the software.

Tommy McClain

 

[Shifty Geezer]

It's something that I could see them do. Just not sure they would go with needing the TrackClip Pro attachment. I guess they could sell specially designed headsets that have the TrackClip Pro integrated in.

Watching the first half of that vid to see what it is, it looks like the headpiece is just 3 IR LEDs on a frame for triangulation.

Kinect's CCDs are at 640x480 too and they claim a 57° FOV. Just not sure about the sample rate & response time. But it looks like the Kinect hardware is probably technically capable of doing everything TrackIR5 can do. The rest is all up to the software.

Framerate would be an issue. We know they have a 30 hz limit, and I imagine the top speed if that's a software cap is 60 fps. Probably still fast enough. Of course this couldn't work with depth perception simultaneously.

 

[patsu]

MAG though is multi-player only, so voice recognition is pointless imho - you talk to real people so you don't need the game to know what you are saying, unless perhaps you want it to automatically map those commands into graphical feedback like waypoints and routes. But then I think you'd be better off with advanced strategy interface in the manner that I think you and Shifty discussed earlier (and which I agree would be really, really cool - combine RUSE with MAG, so to speak).

Yes, I was imagining a collaboration interface in MAG. They can probably "cheat" in a SP squad game by delaying enemy movement. As I mentioned, the result will depend on the game design.

Being able to control your arm to make gestures about what to do when voice chat is not available because the enemy would overhear you could be really cool also, but not easy to do with either controller as neither does fingers really well. Would love to see developers try anyway though - even a menu with predefined gestures would be neat. I'm all for anything that makes online gaming more strategic and less run-and-gun but I may be the ony one.

That's true. In KZ2 and MAG, the enemies can hear you if they are nearby. Would be more practical for a slower paced game.

Should want anyway.

The new controllers are pushing new gameplay experiences. IMHO, it is very important to use them in the right way to avoid giving the users a bad impression.

 

[Silent_Buddha]

That'd work very well. Better than Move, as it'd be more realistic. For shooting though, I think they'd be looking at a gun perihperal that'd need a way to be tracked more accurately than currently possible, and that's where a specific controller would come in. To compete toe-to-toe with Move/Wii, a pointer device looks necessary.

This just made me think of something interesting. Since Kinect features an IR (or near-IR) emitter, it has the option to use either a Move like pointing device (camera see's controller) or a Wii like pointing device (controller camera see's IR emitter).

While it wouldn't have the triangulation ability of the Wii controller it does have Kinect's depth camera to provide relative positioning data.

I'm still not sure whether MS will go with a general pointing device or if they'll instead try to evolve Kinect to more precisely track player body parts. I'd imagine they are toying with both options (and more) in their research labs.

Regards,
SB

 

[AzBat]

Patsu, just a reminder we are in a Kinect thread. Thank you.

Tommy McClain

 

[patsu]

The points apply to 360 games nonetheless. It is important for them to perfect the initial impressions. I believe Microsoft understands this very well. This is why they chose to prioritize speech recognition at launch even if some gamers may make noise. Combining voice, gestures, and other unproven schemes will only complicate matters (Some countries don't support native language speech recognition for a start). Should focus on proving/establishing Kinect's strength.

 

[AzBat]

Watching the first half of that vid to see what it is, it looks like the headpiece is just 3 IR LEDs on a frame for triangulation.

Yes, but could Microsoft do that all in software without the attachment?

Framerate would be an issue. We know they have a 30 hz limit, and I imagine the top speed if that's a software cap is 60 fps. Probably still fast enough. Of course this couldn't work with depth perception simultaneously.

Why would depth perception be a problem? I can understand framerate issues, but at 30fps it still could be workable. Maybe not as high precision, but still workable.

Tommy McClain

 

[Shifty Geezer]

Yes, but could Microsoft do that all in software without the attachment?

Not without 3 sampling points. But then visual headtracking using eyes and mouth is probably enough. We'll get an example of that in GT5.

Why would depth perception be a problem? I can understand framerate issues, but at 30fps it still could be workable. Maybe not as high precision, but still workable.

It's quarter resolution with 3D for one thing ,so tracking three points at distance, they'll cover a small window of pixels and not give much resolution to play with. Then there's the IR noise being projected clashing with the IR dots on the headset which will likely mess up the sub-pixel resolution, although they could perhaps filter the dots out as the pattern is known. But then the IR bounceback isn't known, which means the dot will be getting variations as you move, and I can't see how you could calibrate for that.

 

[grandmaster]

Headtracking in GT5 was a bit of a disappointment. Well, a lot of a disappointment.

In the E3 demo, it doesn't track the angle of your face, instead you physically need to move your head up, down, left and right in order to get the view of the cockpit to move likewise.

So a bit pointless, unfortunately and a world away from a VR-style "looking around" the cabin, or looking towards upcoming corners etc.

 

[Silent_Buddha]

Damn, I wonder if it's the distance from Camera that's made it difficult/impossible for the GT5 team to do it like some of the PC solutions using regular webcams.

I'm guessing with the distance involved, the camera just can't reliably distinguish the small head movement involved in looking at one side of the screen or the other. Kinect would presumably have the same problem. If so, that's hugely disappointing.

Regards,
SB

 

[Arwin]

Headtracking in GT5 was a bit of a disappointment. Well, a lot of a disappointment.

In the E3 demo, it doesn't track the angle of your face, instead you physically need to move your head up, down, left and right in order to get the view of the cockpit to move likewise.

If you are right about this, then I understand that a lot of impressions of E3's head-tracking were so negative, as most people reported the head-tracking to only support looking left and right and the sensitivity for it was very bad, often triggering the same kind of 90 degree style 'look left/right' that's in there now if you press left/right the d-pad on a Logitech steering wheel for instance (you can map it onto the DS3 as well but I don't think it is on there by default, which is why few people know this feature was in there already since first Prologue). This is also how OPM described the feature.

So a bit pointless, unfortunately and a world away from a VR-style "looking around" the cabin, or looking towards upcoming corners etc.

There's a suggestion that it will be full head-tracking all around the cabin eventually, but I have my doubts too that it will work great, if even at all. I'm sure we'll find out before long though.

 

[patsu]

The Official Playstation Magazine talked about it in July:
http://www.computerandvideogames.com/article.php?id=256377

'Indeed, there was only one bone of contention: 'Gran Turismo 3D debuted impeccably but the game's face tracking fared less well,' it reports. 'Face-tracking using the PS Eye works on the 200 cars with cockpits and lets you look around by tilting your face away fro the screen, but the early demo we played left us underwhelmed.

...

A slight glance to the left or right is translated by the game after a considerable delay as a full 90-degree crane of your neck. Hopefully Polyphony can smooth things out before release.'

EDIT:
Regarding turning vs leaning, the DF interview with Dr. Marks has an intriguing remarks:

Digital Foundry: The head-tracking we're seeing in some of the games is intriguing. How does it work? Can it only track the physical movement of the head, or can it pick up actual changes to the direction you're facing?

Richard Marks: The head-tracking is not part of Move, it's part of PlayStation Eye libraries we provide. There are many ways and flavours we can use it. We have detection, where it just finds a face. We have things like smile detection, things like that. We also have tracking and lots of different flavours of tracking. The London studio created their own, for example, they have a lot of experience with EyeToy. Everyone always thinks that head-tracking would be the ultimate thing but when you're playing a game you rarely look away from the TV.

Digital Foundry: You'd need to constantly changing the direction of your face, sure. You'd need to be consciously doing that as part of the control scheme.

Richard Marks: We found that although at first you might think looking left and right is more natural, actually leaning left and leaning right works better. If you hook up head-tracking to a first-person shooter, you can lean around a corner. You can peek out and peek in. We've been looking into that for games where there's cover you can hide behind. It's a really neat feeling.

I suspect it's awkward for people to turn their head and keep their eyes focused/fixed on the screen at the same time. I remember there was a video showing turning angle of face from standing distance. Not sure if it applies to people 4-6 feet away though.

 

[Shifty Geezer]

Triangulating a face may not be possible at the resolutions these cameras have available and the distance of the players. If they avoid head rotation and consider tilt and craning, it should work well. For a racing game, players often lift their head up when approaching a hill to see over, and that could be easily tracked and imlpemented. In essence the screen is treated as a window into a 3D world, and the unconscious interfacing has the player position themselves to change view into the window. I think visual-tracking based systems should be based on this principle, rather than trying to map user movements directly to camera control. In essence, subjective control where the player's movements reflect their attempts to gain information on a subject (like road direction or enemy whereabouts), instead of objective control where the player is thinking about controlling an object (in game camera).

Have we seen much by way of optical systems on Kinect? I think all we've seen is background removal and image embedding. Obviously MS want the focus on their USP, but I hope devs don't ignore other possibilities. They have all the data to play with. And to be fair on them, some are, forgoing MS's skeleton tracking systems for their own. I'm sure some developers will incorporate good head tracking (actually easier in Kinect with 3D than Move). Hmm, I hope developers will! There've had ample opportunity with camera peripherals out since day 1 this gen, but no-one's bothered.

 

[patsu]

I think the developers have already brainstormed and prototyped different ideas and implementations since the reveal. The ideas probably came pretty quickly (Remember the finger game ?).

Since Kinect uses the GPU for computation, I believe MS may have to do a "round-trip" implementation first (from DashBoard to Kinect game and back) before third party can follow similar path to implement their own (complex) tracking.

In the short term, we should see "simple" customization on top of MS's camera methods. MS can proceed to stack supporting services like speech recognition on top of the camera work. They just need more time to polish and integrate the system. They are more savvy in this sense (Sony takes the piece-meal/incremental approach and will most likely deliver Move without good integration and user experiences).

 

[Billy Idol]

Kinect Manual:

http://www.vg247.com/2010/10/05/entire-kinect-manual-photographed-play-distances-power-details-more/

 

[AzBat]

The only thing really new is the confirmation of the Wi-Fi extension cable included with the sensor. They want you to keep your Wireless Adapter mounted to the back, but want you to disconnect it from the rear USB port & use the extension cable to connect it to the front USB ports instead. This frees up the rear USB port to be used for the Kinect sensor.

Tommy McClain

 

[eastmen]

Yikes so many people are down about how far away oyu need to be from the kinect. I don't get it , how close are people who are playing the wii or move ?