Head Tracking for Desktop VR Displays using the WiiRemote & PS Eye *

[DieH@rd]

Nextgen has arrived with Wii.

Using the infrared camera in the Wii remote and a head mounted sensor bar (two IR LEDs), you can accurately track the location of your head and render view dependent images on the screen. This effectively transforms your display into a portal to a virtual environment. The display properly reacts to head and body movement as if it were a real window creating a realistic illusion of depth and space. By Johnny Chung Lee, Carnegie Mellon University. For more information and software visit http://johnnylee.net

I cant belive that such belivable 3d effect can be achieved with NO PENALTY TO HARWARE. Its just smart camera effect, that can be implemented in every game. Damn, i now i want to see this 3d rendering in every MGS cutscene. Can someone try this at home, all software can be found on his site.

Is there some freeware game with open source engine that can be mofidied to use this feature. Maybe Quake 2?!

Discuss.

 

[Acert93]

I saw this last night--the guy has a lot of nice stuff. The general idea isn't totally new, though, as TrackerIR has had a commercial product doing the same stuff for a couple years. The difference? It is $200 and works with only a few PC games.

When he adds the 2nd Wiimote at the end is when it gets interesting. With a large screen HDTV you can see how an FPS with cover system could really come about with motion recognition. 2012 cannot come soon enough

 

[Shifty Geezer]

The problem with head tracking is you turn away from the screen if you're doing AR type positioning. If you're just measuring ducking and diving, standard camera tech out to suffice already. Face recognition is readily capable of tracking a player's phizzog.

I also wonder how cameras could work with IR, seeing as CCD's are often sensitive. Video cameras can see the IR emissions of a remote control. If PSEye and Live! Camera can see IR points are areas of brightness, you could do the same thing with them.

 

[Nesh]

Nextgen has arrived with Wii.

I cant belive that such belivable 3d effect can be achieved with NO PENALTY TO HARWARE. Its just smart camera effect, that can be implemented in every game. Damn, i now i want to see this 3d rendering in every MGS cutscene. Can someone try this at home, all software can be found on his site.

Is there some freeware game with open source engine that can be mofidied to use this feature. Maybe Quake 2?!

Discuss.

That guy does generally amazing things, He found many usages with the wiimote as well as other things!! He is unbelievable!!

 

[epicstruggle]

I also wonder how cameras could work with IR, seeing as CCD's are often sensitive. Video cameras can see the IR emissions of a remote control. If PSEye and Live! Camera can see IR points are areas of brightness, you could do the same thing with them.

They usually see them as purple points of light. Unless you can guarantee that your not going to have purplish colors in your scene, you should be able to rig something like that.

As has been mentioned in this thread, this solution is only note worthy because it is relatively cheap (and uses a wii). This type of tracking has been done before, and because of its limitations isnt really useful except for limited applications. One way to get around it, is by using multiple cameras/wiimotes around the rooms. At this point you might as well just stick the camera on your head to do the tracking, or use magnetic field generators and sensors (what I used). THe only problems is cost.

 

[Shifty Geezer]

The other question is how useful would the implementation actually be? The example vid looks great, but is quite limited in application I think. The obvious use would be FPSes, but requiring players to actually duck or sidestep is asking too much from the gamer, and if the effect 'works' to involve the player, they'll naturally want to look around the TV. Outside of that, would it have benefit in sports games, racers, or other action games? Perhaps. Racers seems ideal as you naturally fix your position looking forward, and being able to follow the driver's head craning to look over hills and round corners would be extremely realistic. If so, I expect some $200 apparatus for GT5 to be released! It would also work well with party games, except that they're to be played with more than one person on a system which the technology can't cope with. It could add immersion to adventure games like Blue Dragon/FF with the added sense of 3D from parallax effects, but not really any gameplay advances.

 

[digitalwanderer]

This might have a few technical issues still, but that still doesn't take away from the "kewl factor" for me...this is neat!

That guy is amazing, pure geek mojo.

 

[epicstruggle]

The other question is how useful would the implementation actually be? The example vid looks great, but is quite limited in application I think. The obvious use would be FPSes, but requiring players to actually duck or sidestep is asking too much from the gamer, and if the effect 'works' to involve the player, they'll naturally want to look around the TV. Outside of that, would it have benefit in sports games, racers, or other action games? Perhaps. Racers seems ideal as you naturally fix your position looking forward, and being able to follow the driver's head craning to look over hills and round corners would be extremely realistic. If so, I expect some $200 apparatus for GT5 to be released! It would also work well with party games, except that they're to be played with more than one person on a system which the technology can't cope with. It could add immersion to adventure games like Blue Dragon/FF with the added sense of 3D from parallax effects, but not really any gameplay advances.

I think you hit it pretty much dead on center. AR/VR only gets interesting once your more mobile with a display that "moves" with you.

 

[habbe]

so the wiimote sitting at the Tv set is tracking 2 infrared diods at the guys head.... A regular webcam can do that. Pretty cool.

 

[epicstruggle]

so the wiimote sitting at the Tv set is tracking 2 infrared diods at the guys head.... A regular webcam can do that. Pretty cool.

It could, but you would need to eliminate all purplish color from your room, clothes. It is not an ideal way to do it.

 

[Altivec]

It could, but you would need to eliminate all purplish color from your room, clothes. It is not an ideal way to do it.

um, not quite. Most webcams can be made to only see IR light, there are a few videos on youtube explaining how to do this. Quite a few cams can even see IR light stock, with no filters removed from the device.

 

[Skrying]

The real advantage of using the Wii is that it was handy, could be easily switched around for this functionality in games (sorta), and that it has a relatively high resolution for the task.

 

[epicstruggle]

um, not quite. Most webcams can be made to only see IR light, there are a few videos on youtube explaining how to do this. Quite a few cams can even see IR light stock, with no filters removed from the device.

Ive been out of the game for a while, so things could/probably have changed. I wasnt aware that new webcams could do it without a filter. Cool.

 

[Npl]

so the wiimote sitting at the Tv set is tracking 2 infrared diods at the guys head.... A regular webcam can do that. Pretty cool.

Im not sure how the wiimote works in detail, but I doubt a regular webcam would work.
If you only have 2 points of reference, how do you know the position in Space? If the two points move toward each other (on a line), it could mean that:
* User turned his head right
* User turned his head left
* User stepped back.
or a combination of stepping back + turning head.

Without measuring distance in some way this cant ever work.

 

[StefanS]

Im not sure how the wiimote works in detail, but I doubt a regular webcam would work.
If you only have 2 points of reference, how do you know the position in Space? If the two points move toward each other (on a line), it could mean that:
* User turned his head right
* User turned his head left
* User stepped back.
or a combination of stepping back + turning head.

Without measuring distance in some way this cant ever work.

Well, that's all the Wiimote really does. It reads two infrared sources' location off the "sensor" bar. Combined with its accelerometers it can calculate a direction.

Basically what the demo does here is just basic trigonometry: If the distance between the infrared sources is known, you can deduce the distance from the screen assuming that the user doesn't move at superhuman speeds: i.e. if the distance of the sources decreases too rapidly between two (or possibly more) reference time frames you can fairly certainly deduce he's moving his head.

Also assuming a standard living room size (5meters depth), the separation of the sources would still be ~2.3 degrees (as seen by the wiimote). Reduce that to 2 meters like in the demo and it grows to ~5.7 degrees. So anything below that threshold is automatically deduced to be head turning.

Combining both approaches should work pretty well I suppose.

Moreover, - and that is more of a speculation [I've got zero clue how sensitive the IR camera really is] - he could try to measure the intensity of the sources:Correspondence meaning head not turned, person is moving away. Noncorrespondence meaning the person is turning his head. Comparing that to different reference timeframes would be an even more powerful solution (giving additional complementary information [apart from the velocity calculated from trigs and reference timeframes] on turning speeds, distance from the source).

I am sure there're lots of other ways to do it, but those are just the ones I could come up in 5 mins

 

[Npl]

Well, that's all the Wiimote really does. It reads two infrared sources' location off the "sensor" bar. Combined with its accelerometers it can calculate a direction.

Except the Wiimote is fixed in that example.

Basically what the demo does here is just basic trigonometry: If the distance between the infrared sources is known, you can deduce the distance from the screen assuming that the user doesn't move at superhuman speeds: i.e. if the distance of the sources decreases too rapidly between two (or possibly more) reference time frames you can fairly certainly deduce he's moving his head.

deducing hes moving his head is no prob - but detecting HOW hes moving (left or right if we rule out backwards?). Head turning left would produce the same information as Head turning right - if everything you got is 2 reference points projected on a plane.

Combining both approaches should work pretty well I suppose.

The problem is that both approaches (I assume you mean measuring distance assuming the heads not turned and measuring angle assuming the distance is known) are in confict?

...snip...

There are surely several ways to solve those problems, but I cant think of an easy one using a webcam.

 

[Kryton]

Except the Wiimote is fixed in that example.
deducing hes moving his head is no prob - but detecting HOW hes moving (left or right if we rule out backwards?). Head turning left would produce the same information as Head turning right - if everything you got is 2 reference points projected on a plane.
The problem is that both approaches (I assume you mean measuring distance assuming the heads not turned and measuring angle assuming the distance is known) are in confict?

There are surely several ways to solve those problems, but I cant think of an easy one using a webcam.

I think he is using lateral movement to adjust the angle. When he's walking through and demonstrating the range of movement, the sensor bar is never angled - just moved through the 6 dimensions. There isn't any way of doing it with 1 webcam and 2 identical light sources (you could pulse them, use a coloured LED next to the IR one and track colour too, ...)

 

[Npl]

I think he is using lateral movement to adjust the angle. When he's walking through and demonstrating the range of movement, the sensor bar is never angled - just moved through the 6 dimensions. There isn't any way of doing it with 1 webcam and 2 identical light sources (you could pulse them, use a coloured LED next to the IR one and track colour too, ...)

That makes sense, except the 6 dimensions bit

 

[Shifty Geezer]

Im not sure how the wiimote works in detail...

The Wiimote works exactly using a 'webcam'. There's no distance measuring tech or anything fancy in there. It's just a CCD array detecting IR hotspots and then software to translate that. Thus anything you can do with Wiimote's IR camera, you can do in principal with any other camera device as long as you can successfully obtain points of reference. In a game situation you can also add the motion forces values to your calculations, but in this implementation the camera is static so they're no good for head tracking.

 

[StefanS]

Except the Wiimote is fixed in that example.

Of course, didn't say otherwise, Just said how it is normally done

deducing hes moving his head is no prob - but detecting HOW hes moving (left or right if we rule out backwards?). Head turning left would produce the same information as Head turning right - if everything you got is 2 reference points projected on a plane.

Nope, the Wiibar is not rotating around its own centre, but around the rotation axis of the persons head, causing a detectable offset. Thus the direction of turning is easy to detect.

The problem is that both approaches (I assume you mean measuring distance assuming the heads not turned and measuring angle assuming the distance is known) are in confict?

You're misunderstanding the approach here. It's complimentary: If the points are closer than a certain threshold[it's a restriction,i.e. he can't be as far away as x metres], he's turning his head (the angle [dot1-wiimote-dot2] is rather an angular resolution which is used to calculate the minimal distance possible for the IR dots recorded by the camera.).
E.g.: x > minimal distance -> turning or movement
x< minimal distance -> only turning

There are surely several ways to solve those problems, but I cant think of an easy one using a webcam.

Oh my approach would only need a camera sensitive enough to detect the intensity differences and very straightforward image processing nothing to fancy. As long as the camera is sensitive enough it'll work.