Softkinetic DepthSense® Human Tracking Library for PS4™

[mosen]

We have a video showing seemingly low latency when you watch the guy with the sword kicking blocks.

That's a marketing video and we don't even know if they are showing PS4 results or not. I can see one of their ToF cameras (this one, with 100$ higher price than Kinect v2 for Windows at lower specs) at 1:59 (on top of the TV) in the same video that was supposed to be about their tech on PS4.

I think the depth image that they shown around 1:57 (which has low res/quality) or the gameplay that they shown at the beginning of the video (which has low quality polygonal masking) seems to be the real output from the PS4 camera.

It shows a very obvious lag between her movements and the avatar. It's most perceptible when she waves her arms up and down while walking side to side. Her arm is up when the avatar's arm is pretty much down.

I saw similar lag with PS Move and the explanation was that it's display latency not controller, you can compare RGB output with Depth output (probably) to measure the latency.

http://www.eurogamer.net/videos/digitalfoundry-playstation-move-latency-analysis

 

[Shifty Geezer]

That's a marketing video

It's an official video from them selling their library for PS4. If they aren't using the PS4 system, it's false representation. The guy's holding a Move, so I see little reason to question it's a PS4 implementation.

I saw similar lag with PS Move and the explanation was that it's display latency not controller, you can compare RGB output with Depth output (probably) to measure the latency.

http://www.eurogamer.net/videos/digitalfoundry-playstation-move-latency-analysis

That's a fair point. Still, the video was showing OnQ's idea of latency being an issue, which it did show, just, probably, incorrectly.

 

[onQ]

Perhaps we're butting heads against choice of words again? Whether Kinect is 30 fps or 60 fps, I expect the game design choices to remain the same. I don't see anything one could do at 60 fps based on arm waving head wobbling that wouldn't also work at 30 fps, only less well. Things like sword fighting are more timing based than twitch based, because you see the opponent's movements well ahead of when they strike and intercept. You don't want too much lag, of course, but that sort of game style won't be enabled at 60 fps and impossible at 30 fps.

Yes, it makes it more sluggish I'll agree, although it's not proven that running the IR feed on Kinect at 60 fps will result in much smoother tracking. There's a lot of processing overhead. Switching to 60fps may just mean an 17-33 ms saving latency on each frame delay. That's also true of SoftKinect. Latency will be more a matter of how much needs to be buffered.

Basically, the 30/60 fps input argument is the same as the 30/60 fps output argument. Higher framerates are smoother, but don't generally open up new gameplay types. Lower latency could make some game styles that were tried unsuccessfully on other systems more palatable for consumers though.

I don't see PS4 running 60 fps opening up more game styles to motion gaming. It'll be the same set of game styles, only a bit smoother if the tracking tech is as good as it promises (which is rarely the case).

It's not that it opens up the door for gameplay that couldn't be done at 30 fps it's the fact that it make the game more enjoyable and worth playing. Devs already know that the smoothness of the controls make a difference in how their game will play so we might see devs making 60 fps games that they wouldn't want to create with 30 fps tracking.

Kinect 1 was also 60fps capable on PC.

I didn't say lower price leads to lower quality, but stereo vision cameras have lower quality than ToF cameras (in this case it has lower quality than even Kinect 1). They also need more compute resources to operate and you can't compare their solution with Kinect 1/2 since we know nothing about delay/latency of their solution.

Kinect 2 end to end latency is 34ms. Less than 14 ms to sends a packet of data (the color image, depth data, and sound) plus 20ms latency to software. Do you have similar numbers for PS4 camera?




This video shows nothing, she tends to wave her hands slowly to show smooth tracking capability of the new Kinect which is what they want to do. I played many Kinect 1 games and I didn't need to adjust my movement with avatars.

Even if it does use more processing the PS4 has over 500GFLOPS of extra compute free compared to the Xbox One that could be used for motion tracking.
GPGPU's/APU's are designed to handling computer vision. My old laptop was able to handle 125FPS hand tracking with the PS Eye using it's weak CPU so I'm sure that the PS4 can handle 60FPS tracking using the hardware that was built for it like AMD APUs.


By the way you must be really slow if you're not adjusting yourself for 30 fps tracking when playing a game when you have to jump around swinging your arms or legs. You're trying to tell me that 30fps is good enough for arm swinging when 100+fps is preferred for good head tracking? If you can see your movements on screen chances are you will adapt to them.

 

[onQ]

That's a marketing video and we don't even know if they are showing PS4 results or not. I can see one of their ToF cameras (this one, with 100$ higher price than Kinect v2 for Windows at lower specs) at 1:59 (on top of the TV) in the same video that was supposed to be about their tech on PS4.

I think the depth image that they shown around 1:57 (which has low res/quality) or the gameplay that they shown at the beginning of the video (which has low quality polygonal masking) seems to be the real output from the PS4 camera.



I saw similar lag with PS Move and the explanation was that it's display latency not controller, you can compare RGB output with Depth output (probably) to measure the latency.

http://www.eurogamer.net/videos/digitalfoundry-playstation-move-latency-analysis

The sword fighting & cherry game clearly show that they are using the PS4 controller & PlayStation Move. You see their camera only in the part about that stunt game or whatever that is, they also show Just Dance being played using a phone does that mean that the sword fighting demo is a phone app?.

The video says that the PS4 Human Tracking Library 59 fps command buffer is 3ms. ( 2.6447ms from the CPU & 0.3987ms from the GPU)

 

[Shifty Geezer]

It's not that it opens up the door for gameplay that couldn't be done at 30 fps it's the fact that it make the game more enjoyable and worth playing. Devs already know that the smoothness of the controls make a difference in how their game will play so we might see devs making 60 fps games that they wouldn't want to create with 30 fps tracking.

Okay, although originally you said:

What I'm saying is that if the camera was tracking at 60FPS or higher it would open the door for more Kinect games because devs could make games that use twitch gameplay.

'Twitch gameplay' still isn't an option because large motions are intrinsically slow. 30 fps input and 60 fps on screen with interpolated movement would do just as well for high-speed motion gaming. Of the motion games and ideas available, lower latency could make them more appealing.

 

[onQ]

Okay, although originally you said:
'Twitch gameplay' still isn't an option because large motions are intrinsically slow. 30 fps input and 60 fps on screen with interpolated movement would do just as well for high-speed motion gaming. Of the motion games and ideas available, lower latency could make them more appealing.

Say you're playing a game where you have to jump up , duck & so on as objects come at you would you rather the game get 5 frames of data from your jump or 10 frames of data?

 

[Shifty Geezer]

Doesn't make much difference as the gameplay is timing based. Human reactions are poop and we could never rely on them to respond to complex gaming events without preparation. With enough forewarning, you can execute a jump command in time to execute the jump, even if that jump comes a half-second after pressing. Getting used to lag adds a barrier to entry, but it's easily accommodated.

You need enough data to determine the player's jumping. That'll required a noted upwards acceleration, or the head being above standing height or something. These sorts of movement tracking events require enough information to make the right interpretation, which isn't a count of frames rather than an amount of movement. eg. Let's say you want to tell if the player is performing a shield bash or a raising their shield to block. You'll be able to tell the shield is moving the moment the hand starts, but you'll have to wait until the block position is reached 1/10th second later to see if the hand then continues past that point or stops. No matter what framerate that camera, it's going to have to wait until the arm has gone through enough motion to read it correctly.

Moving away from gesture based gaming and to direct control would help solve some issues, but then we won't be talking about 5 frames or 10 frames to execute a jump. The game will just match where you are into the game frame by frame. Interpolation could probably fit a 30 fps nicely into a 60 fps game, although that'll add latency.

 

[mosen]

It's an official video from them selling their library for PS4. If they aren't using the PS4 system, it's false representation. The guy's holding a Move, so I see little reason to question it's a PS4 implementation.

You can use PS Move on PC, too. That part of the video really shows nothing to me.

Even if it does use more processing the PS4 has over 500GFLOPS of extra compute free compared to the Xbox One that could be used for motion tracking.
GPGPU's/APU's are designed to handling computer vision. My old laptop was able to handle 125FPS hand tracking with the PS Eye using it's weak CPU so I'm sure that the PS4 can handle 60FPS tracking using the hardware that was built for it like AMD APUs.


By the way you must be really slow if you're not adjusting yourself for 30 fps tracking when playing a game when you have to jump around swinging your arms or legs. You're trying to tell me that 30fps is good enough for arm swinging when 100+fps is preferred for good head tracking? If you can see your movements on screen chances are you will adapt to them.

PS4 has more FLOPS but low latency is more important than higher FLOPS for this kind of tasks, according to XB1 architects Exemplar processing doesn't need much ALUs and it's more latency sensitive. XB1 should have less or equal (at worst) latency for this propose.

I didn't say 30fps is enough, 60fps will be much better but I had no problem with knowing what I am doing in front of Kinect. I don't need to see my avatar to choose what I need to do, I do what I want and may avatar will follow me accordingly. There is no need to mach yourself with your avatar.

The sword fighting & cherry game clearly show that they are using the PS4 controller & PlayStation Move. You see their camera only in the part about that stunt game or whatever that is, they also show Just Dance being played using a phone does that mean that the sword fighting demo is a phone app?.

The video says that the PS4 Human Tracking Library 59 fps command buffer is 3ms. ( 2.6447ms from the CPU & 0.3987ms from the GPU)

You need to see that video again. There is a massive difference between character masking at the beginning of the video and what you can see with PS Move section of the video. Just think of it. The character masking was horrible at the start of the video and the other one was just like what they showed with their ToF camera and Kinect 2 (more or less). Night and day difference.

You may like to see their actual demo with PS4/PS4 camera in this video (from 1:33-1:44):

First, you can see their demo (that uses Nvidia SHIELD and their own ToF camera) that is what they shown at PS4 video (flying character and the avatar that they used for PS Move demo). After that you can see their masking problems, latency between RGB data and skeletons output and a kind of floating/slow-motion movements (without considering display latency which we can't be seen from this video). There is no sign of a 60fps camera on their tech-demo.

 

[Shifty Geezer]

You may like to see their actual demo with PS4/PS4 camera in this video (from 1:33-1:44):

As dreadful as I expected.

 

[onQ]

You can use PS Move on PC, too. That part of the video really shows nothing to me.



PS4 has more FLOPS but low latency is more important than higher FLOPS for this kind of tasks, according to XB1 architects Exemplar processing doesn't need much ALUs and it's more latency sensitive. XB1 should have less or equal (at worst) latency for this propose.

I didn't say 30fps is enough, 60fps will be much better but I had no problem with knowing what I am doing in front of Kinect. I don't need to see my avatar to choose what I need to do, I do what I want and may avatar will follow me accordingly. There is no need to mach yourself with your avatar.



You need to see that video again. There is a massive difference between character masking at the beginning of the video and what you can see with PS Move section of the video. Just think of it. The character masking was horrible at the start of the video and the other one was just like what they showed with their ToF camera and Kinect 2 (more or less). Night and day difference.

You may like to see their actual demo with PS4/PS4 camera in this video (from 1:33-1:44):

First, you can see their demo (that uses Nvidia SHIELD and their own ToF camera) that is what they shown at PS4 video (flying character and the avatar that they used for PS Move demo). After that you can see their masking problems, latency between RGB data and skeletons output and a kind of floating/slow-motion movements (without considering display latency which we can't be seen from this video). There is no sign of a 60fps camera on their tech-demo.

You might be right about them using their own camera in the PlayStation Move demo but it still doesn't prove that PS4 camera won't play the same way when PlayStation Move is being used with inverse kinematics.

 

[onQ]

Sony bought Softkinetic


http://www.sony.net/SonyInfo/News/Press/201510/15-083E/index.html

Sony Acquires Belgian Innovator of Range Image Sensor Technology, Softkinetic Systems S.A., in its Push Toward Next-Generation Range Image Sensors and Solutions

Tokyo, Japan - Sony Corporation ("Sony") is announcing that it has completed the acquisition of Softkinetic Systems S.A. ("Softkinetic"), after reaching an agreement with the company and its major shareholders. With this acquisition, Softkinetic - which possesses time-of-flight ("ToF") range image sensor technology, as well as related systems and software - has become a wholly-owned subsidiary of Sony.

ToF is a method for resolving the distance to an object. ToF distance measurement pixels, which are laid on top of the sensor in two dimensions, measure the flight time (delay) it takes for light to leave the light source, reflect off the object, and return to the image sensor.

Sony will focus on combining Softkinetic's ToF range image sensor technology expertise with its own technologies with the aim of developing the next generation of range image sensors and solutions, not only in the field of imaging, but for broader sensing-related applications as well.

No material impact is anticipated on Sony's consolidated financial results for the fiscal year ending March 31, 2016 as a result of the acquisition.
Overview of Softkinetic Systems S.A.
Headquarters:
Brussels, Belgium
Employees:
77
Business offices:
Brussels, Belgium; Sunnyvale, CA, USA
Primary areas of business:
Development and licensing of ToF range image sensors
Development and licensing of range imaging software, such as for range image signal processing and gesture recognition
Development of ToF range image sensor modules and reference designs

 

[Shifty Geezer]

Reads more like acquiring essential imaging IP for Sony's imaging sensors and staying ahead in the business they dominate. These sensors may then appear in headsets etc. for Sony Computer Entertainment, but it's not really aimed at that. Expect Sony TOF sensors in future devices everywhere, like their ubiquitous cameras. Maybe even Hololens if the tracking is that effective and the part commodity priced.

 

[onQ]

Reads more like acquiring essential imaging IP for Sony's imaging sensors and staying ahead in the business they dominate. These sensors may then appear in headsets etc. for Sony Computer Entertainment, but it's not really aimed at that. Expect Sony TOF sensors in future devices everywhere, like their ubiquitous cameras. Maybe even Hololens if the tracking is that effective and the part commodity priced.

I wouldn't ignore the fact that they also get the software lib.

Plus VR.

 

[Shifty Geezer]

Sure, but that wasn't the major reason for the purchase. Providing people with sensors, it's also in Sony's interests to provide software to use those sensors to aid adoption. Unlike other sensor providers, for example, who just give a device, Sony can give a complete solution. It's nothing particular to consoles though. I think that's very apparent from the lack of SoftKinect in PSVR. It's not like Sony had an interest in this tech for VR ahead of the acquisition (probably far too costly for now). It'll no doubt make an appearance years from now, but we'll have TOF cameras in iThings before that.

 

[onQ]

Sure, but that wasn't the major reason for the purchase. Providing people with sensors, it's also in Sony's interests to provide software to use those sensors to aid adoption. Unlike other sensor providers, for example, who just give a device, Sony can give a complete solution. It's nothing particular to consoles though. I think that's very apparent from the lack of SoftKinect in PSVR. It's not like Sony had an interest in this tech for VR ahead of the acquisition (probably far too costly for now). It'll no doubt make an appearance years from now, but we'll have TOF cameras in iThings before that.

The Softkinetic software work with more than just Time-of-Flight cameras.

 

[Arwin]

I think it was interesting as well to hear how successfully the optics part of Sony contributed to the VR headset.