Stereo Vision and Time of Flight setups for Face Scanning

[Rurouni]

I'm hoping for depth without any overlay just to see how sensitive Kinect 2 TOF solution towards depth. With overlay, what I see is the overlay image and not the depth itself. For stereo cam approach, the more closer the object to the cam, the more depth info you can get on the object. The depth resolve is tied to the cam res. But TOF use the timing of the light, thus near or far shouldn't affect the depth precision too much vs stereo approach.

 

[onQ]

I'm hoping for depth without any overlay just to see how sensitive Kinect 2 TOF solution towards depth. With overlay, what I see is the overlay image and not the depth itself. For stereo cam approach, the more closer the object to the cam, the more depth info you can get on the object. The depth resolve is tied to the cam res. But TOF use the timing of the light, thus near or far shouldn't affect the depth precision too much vs stereo approach.

There is a video with a guy showing the camera feed with the PC Kinect 2.0 I'll try to find it & make screen caps he also said that you couldn't get to close.


Edit: I have them. Seems that there's no way of getting close enough to get better face details with the depth cam even Kinect 1 let you get closer.

 

[Rurouni]

I've just seen the latest hand demo, and from it, you can see that it had trouble to track the hand movement if you just move the finger mainly in the Z direction (apparent on the very last scene with 4 person demoing. The bottom right person making an open-closed hand movement that the Kinect 2 having difficult time picking that up).

I remember one of the demoed Kinect 2 game that have face scanning feature (not face mapping, but the actual shape), and it needed the player to rotate their head which probably means looking for the silhouette of the head instead of relying on the Z resolution (not to be confused with XY resolution) only.



What I'm confused about is that the EA thing should be simple. It doesn't even need depth camera, unless they also wants to have correct topology of your face. Basically whatever magic EA are doing, they are doing something forbidden with it... some kind of black arts like trying to resurrect the dead.

Edit: 2K not EA. Haven't followed any sport games outside of Fifa.

 

[onQ]

A guy on operationsports.com forums just posted a video of his face scan & I have to say this tech is pretty amazing when it work. Look at the small details that it picked up on in his skin.

http://www.operationsports.com/foru...765734-2k15-post-your-facescan-thread-79.html

Screen caps from the video.

 

[orangpelupa]

wow, this for MMO! i totally gonna scan my cat face for my character

 

[zed]

Kinect the device that just keeps on giving

 

[Shifty Geezer]

It doesn't even need depth camera, unless they also wants to have correct topology of your face.

It does. It captures face shape as well as picture to slap on it

wow, this for MMO! i totally gonna scan my cat face for my character

Can you imagine the abomination that will result? The thing can barely match a human head to the human-head model, let alone a cat!

 

[Rurouni]

I mean it doesn't need to capture the face topology, only the outline.

 

[orangpelupa]

@shifty lol i imagine it will be awescary (adapted from kimokawaii japanese )
i love scanning my cat with kinect in kinect fun labs.

btw anybody tested the scanning while wearing glasses? what will happen?

 

[Shifty Geezer]

The recommendations are to remove any confusing elements including facial hair. Glasses should put an image of them on the face, which makes little sense.

 

[onQ]

This one is probably the biggest indication that it's the signal coming in from the camera that's causing some of the problems.

 

[Laa-Yosh]

This all sounds like the stone age to me. We use 36 DSLR cameras and it's still not enough

 

[Scott_Arm]

Tempted to get the game just to see if I can make face scanning on xbox one work.

 

[onQ]

Tempted to get the game just to see if I can make face scanning on xbox one work.

Some people have gotten it to work.


I think we have a situation where the 2 different camera technologies are working against each other.

RGB wants you to get closer to get better detail but the active IR don't want you that close. RGB would love some natural light from the Sun shining in so it can see your face better but too much IR coming from the Sun is causing problems with the active IR.

 

[Laa-Yosh]

Just for fun, this is how our raw face scans look like.

Yeah, that'd be me...

 

[function]

Dood, you have no arms! How do u even mouse?

 

[onQ]

Just for fun, this is how our raw face scans look like.

Yeah, that'd be me...

Photometric or Structured Light? & if you had to guess what would you say is causing the problems with Kinect 2.0 face scans?

 

[Laa-Yosh]

Dood, you have no arms! How do u even mouse?

Not gonna show the full body scans here Face scans use a different arrangement to maximize precision.

Photometric or Structured Light?

It's photogrammetry, we use a 36 camera rig currently. Face scans are reasonably good, full body loses precision but we can compensate by merging multiple partial scans using the full body as a template to combine them.
We'll see if we need to do enough full body scans in the future to justify buying more cameras. Also, it can be combined with structured light, but we'd need several good projectors for that and we don't yet know if that's worth the price vs more cameras. It's still experimental

& if you had to guess what would you say is causing the problems with Kinect 2.0 face scans?

A single shot, even from a stereo setup, is not enough to compensate for the self-occlusion. HD resolution isn't enough for small details either, and of course hair, eyelashes and such will always mess up the results.
Basically, these simple systems lack precision, there's too much uncertainty, but you can still get lucky.

 

[onQ]

Not gonna show the full body scans here Face scans use a different arrangement to maximize precision.



It's photogrammetry, we use a 36 camera rig currently. Face scans are reasonably good, full body loses precision but we can compensate by merging multiple partial scans using the full body as a template to combine them.
We'll see if we need to do enough full body scans in the future to justify buying more cameras. Also, it can be combined with structured light, but we'd need several good projectors for that and we don't yet know if that's worth the price vs more cameras. It's still experimental



A single shot, even from a stereo setup, is not enough to compensate for the self-occlusion. HD resolution isn't enough for small details either, and of course hair, eyelashes and such will always mess up the results.
Basically, these simple systems lack precision, there's too much uncertainty, but you can still get lucky.

OK thanks,

 

[Shifty Geezer]

A single shot, even from a stereo setup, is not enough to compensate for the self-occlusion.

They don't use a single shot.
http://support.2k.com/hc/en-us/articles/203696076-How-to-Scan-Your-Face-in-NBA-2K15

1. You want to turn your head SLOWLY 30 degrees to the left and right as the camera is taking hundreds of pictures of you. Keep your chin on a level field. Don’t move the camera during this process.

That's basically the same as using hundreds of cameras around the face, excepting of course they aren't exposed at the same moment but instead sequentially, during which the head moves. But there's plenty of detail and reference points to construct the rough model.