Softkinetic DepthSense® Human Tracking Library for PS4™

[onQ]

A showcase of games powered by our DepthSense Human Tracking Library (HT-Lib) for PS4™: Just Dance® 2014, Rabbids Invasion: The Interactive TV Show®, Just Dance 2015® by Ubisoft® and Commander Cherry’s Puzzled Journey by Grandé Games.

Get inspired and learn more at www.softkinetic.com/products/htlib

HUMAN TRACKING LIBRARY (HTLIB)
DepthSense® HT-Lib, our multi-user full-body tracking middleware, is now available to all PlayStation®4 (PS4™) developers.

HT-Lib enhances the quality of the depth map and stereoscopic features provided by the PlayStation®Camera for PS4™. It has been developed with performance as the top priority. It's specifically optimized for the PS4™ GPU, to use minimal CPU, and memory resources. It delivers fast, low latency data transfer to create the best user experience on any console platform.

To find out more, contact us directly at sales@softkinetic.com for more details about getting started with DepthSense HT-Lib for PS4™.

KEY FEATURES
Full Body Tracking
Multi-user : 4 player simultaneous tracking
Detailed movements : 21 joints tracked per player
Fast, low latency (30/60 FPS)
User Mask
Clean user cutout
Green screen effect without the set-up or know-how
Scene Analysis
Automatic and transparent scene calibration
Easy filtering of furniture, floor, walls and objects
Complex occlusion management
Global product features
Written in standard C++
Record data as a move file (.SKV format)
Supports the PlayStation®Camera for PS4™ (Kinect and DepthSense® coming soon*)
Plug-in for Unity 4.3 with samples and demos
Available on the PS4™ platform (Android & Windows coming soon*)
Performance & System Architecture
Optimized for PS4™ GPU
Tuned to use minimal CPU and memory resources
Rapid refresh rates at 30/60 fps

*iisu® middleware supports Kinect, DepthSense, Windows, Android & Linux and is available at www.softkinetic.com/products/iisumiddleware

UNITY PLUG-IN
Our DepthSense HT-Lib plug-in is available today for Unity 4.3 for PS4™, and gives developers access to the full range of features inside our human-tracking library.

We also provide a variety of Unity samples and demos to help get you started.

GAME TESTIMONIALS
We created HT-Lib to handle all variety of game types and scenarios. From full-body avatar controls, to analyzing player behavior. From augmenting traditional controller play, or even targeting specific body parts for control. Whatever your gaming needs, we can help make it happen.

Ubisoft® chose HT-Lib to develop Just Dance® 2014 for the launch of the PS4™, and there are many more games in development for release this year.

GAME TITLES: JUST DANCE 2014® & JUST DANCE 2015® (IN DEVELOPMENT)
GAME DEVELOPER: UBISOFT®


"Ubisoft worked closely with SoftKinetic to bring Just Dance 2014® to the PS4, and we are planning to use HTLib for some new titles for the PS4, including Just Dance 2015® and Rabbids Invasion: The Interactive TV Show®."
Martin Oliver, producer, Ubisoft Reflections

GAME TITLE: COMMANDER CHERRY (IN DEVELOPMENT)
GAME DEVELOPER: GRANDE GAMES

"At Grandé Games we have focused on the unique potential of motion control in gaming, and SoftKinetic’s DepthSense HTLib has allowed us to extract the player silhouette in a comfortable and straight forward way in Unity 3D. We are thrilled to be able to realize our game, Commander Cherry’s Puzzled Journey, on the PS4 platform."
Georg Graf, co-founder and programmer, Grandé Games

 

[joker454]

Sweet, now people just has to make games using this. It says this lib was used in Just Dance 2014 which is only so-so as far as human tracking goes, but maybe the 2015 version coming out is using an improved version of their lib, guess I'll find out soon enough. Hopefully the "and there are many more games in development for release this year" part rings true as well.

 

[function]

Looks rather crude compared to Kinect 2, but at least this might get some support ...

 

[joker454]

Looks rather crude compared to Kinect 2, but at least this might get some support ...

Yeah, they had a year to do cool stuff with Kinect 2.0 and didn't so it's time to look elsewhere. Maybe this middle ware will make it easier for devs to hop on board the casual games train in a more financially friendly multi platform manner.

 

[orangpelupa]

the joints seems less than the kinect2 but it is very fast. i did not see any lag

 

[djskribbles]

Looks rather crude compared to Kinect 2, but at least this might get some support ...

I think it has potential and it looks to be good enough for most of the people that these games are targeted to. Not bad at all for a camera that costs almost a 1/3 of Kinect 2.

 

[onQ]

Looks rather crude compared to Kinect 2, but at least this might get some support ...

But for gaming it's better than Kinect 2 just as long as you have good lighting. Kinects advantage is being good in all lighting conditions but it's disadvantage is always tracking at 30 FPS PS4 camera can track up to 240FPS in it's lowest resolution and 60FPS at 1280x800 x 2 Kinect's tracking is limited to 30FPS at 512×424 (not actually 512x424 it's higher but as a depth sensor it's 512x424) while the RBG camera is 1080P

the joints seems less than the kinect2 but it is very fast. i did not see any lag

PlayStation Camera track 21 joints on each person while Kinect track 25 joints on each person. Kinect can track up to 6 people while this tracking library only track 4 people.


fact is Kinect is more reliable in all situations but the PlayStation Camera is better for gaming because 30FPS is always going to feel weighed down when you're controlling a game with your motions. Games like Dance Central and Just Dance is no problem because you're not controlling an avatar so the chunky tracking will never show it's self.

When you add PlayStation Move to the Dual Camera it make Kinect look bad. but Kinect is more fool proof than the PlayStation Camera because it provides it's own array of IR sensors while the PS camera rely on the lighting in your home and off of your TV. Kinect One would be doing so much better right now if they would have pushed for a 60FPS sensor but I guess it would have pushed the price even higher or it was using too much processing or over heating.

 

[temesgen]

Just Dance 2014 for PS4 is a rather poor use of motion controls, its pretty gimmicky. I confident the technology is capable of more but development time or the scope of the title was responsible for the lackluster results.

What middleware does the Playroom use? IMO that game/tech demo is a much better representation of what camera controls are capable of.

 

[Shifty Geezer]

I'm far more concerned about reliability than paper specs. Can it really track 4 people, or is there so much confusion and corruption where players overlap that it's less stable than Kinect 1? Will you need to wear gloves to get reasonable contrast against the light walls? One of the major downsides to Eyetoy was getting it to work. Camera tech has improved so this might be less of an issue, but it's still a concern until the pudding has been proven. Look at the top, 4 player picture of your HTLIB quote. The boundaries of the people are only approximate, with the jeans not mixing well with the carpet. It looks like lots of potential to get confused, but hopefully their skeletons are better implemented than Kinect's and human don't do impossible things with their joints, adding stability.

Not to mention whether anyone uses it or not!

 

[temesgen]

I'm far more concerned about reliability than paper specs. Can it really track 4 people, or is there so much confusion and corruption where players overlap that it's less stable than Kinect 1? Will you need to wear gloves to get reasonable contrast against the light walls? One of the major downsides to Eyetoy was getting it to work. Camera tech has improved so this might be less of an issue, but it's still a concern until the pudding has been proven. Look at the top, 4 player picture of your HTLIB quote. The boundaries of the people are only approximate, with the jeans not mixing well with the carpet. It looks like lots of potential to get confused, but hopefully their skeletons are better implemented than Kinect's and human don't do impossible things with their joints, adding stability.

Not to mention whether anyone uses it or not!

Just Dance struggled to recognize 2nd and 3rd users in my home atleast, I don't think we ever have had it recognize 4 successfully. Playroom does a bit better but there is no doubt that overall lighting in the room as well as what your wearing plays a factor. Overlap is an issue with Just Dance, I don't recall any confusion when my daughter and friends use The Playroom. Maybe not all of the libraries were used in JD2014 for PS4....

 

[steveOrino]

Kinect One would be doing so much better right now if they would have pushed for a 60FPS sensor but I guess it would have pushed the price even higher or it was using too much processing or over heating.

I doubt that. The problem with these set-top motion sensing cameras is the handful of genres they work well with. How many "move limb here" X & Y-axis games are people really interested in beyond what has been done? Would seem not many, especially since an entire generation of consoles have familiarized the gaming population as to how limited these devices are. Games like Fighter Within.. I would probably have an easier time playing Mike Tyson's Punch-out with the Power Glove.

 

[onQ]

I doubt that. The problem with these set-top motion sensing cameras is the handful of genres they work well with. How many "move limb here" X & Y-axis games are people really interested in beyond what has been done? Would seem not many, especially since an entire generation of consoles have familiarized the gaming population as to how limited these devices are. Games like Fighter Within.. I would probably have an easier time playing Mike Tyson's Punch-out with the Power Glove.

What I'm saying is that if the camera was tracking at 60FPS or higher it would open the door for more Kinect games because devs could make games that use twitch gameplay & not worry about the tracking not being as smooth as it should. Motion controls are a lot like touch controls and the speed make a difference a lot of these games that people love on mobile devices now would be hated if they didn't have smooth feedback when you play them.

 

[mosen]

But for gaming it's better than Kinect 2 just as long as you have good lighting. Kinects advantage is being good in all lighting conditions but it's disadvantage is always tracking at 30 FPS PS4 camera can track up to 240FPS in it's lowest resolution and 60FPS at 1280x800 x 2 Kinect's tracking is limited to 30FPS at 512×424 (not actually 512x424 it's higher but as a depth sensor it's 512x424) while the RBG camera is 1080P



PlayStation Camera track 21 joints on each person while Kinect track 25 joints on each person. Kinect can track up to 6 people while this tracking library only track 4 people.


fact is Kinect is more reliable in all situations but the PlayStation Camera is better for gaming because 30FPS is always going to feel weighed down when you're controlling a game with your motions. Games like Dance Central and Just Dance is no problem because you're not controlling an avatar so the chunky tracking will never show it's self.

When you add PlayStation Move to the Dual Camera it make Kinect look bad. but Kinect is more fool proof than the PlayStation Camera because it provides it's own array of IR sensors while the PS camera rely on the lighting in your home and off of your TV. Kinect One would be doing so much better right now if they would have pushed for a 60FPS sensor but I guess it would have pushed the price even higher or it was using too much processing or over heating.

How it's better for gaming than Kinect 2? It has less accuracy than Kinect 1. Higher FPS would be important if it had equal or better accuracy than Kinect 1/2. Also, Kinect 2 is 60 fps capable.

Spoiler

http://forum.beyond3d.com/showpost.php?p=1849343&postcount=272

PS4 camera supports three modes:

1)1280x800p @60fps
2)640x400p @120fps
3)320x192p @240fps

Its quality will be lower at higher FPS, since lower resolution means lower accuracy (you have less pixels to understand object shape/movement from, correctly) and higher FPS needs more compute resources. Also PS4 camera may have some problems with dark-skinned users, ambient lighting and different distances and we know nothing about latency.

So I think it's only cheaper than Kinect 1/2, not better, for motion gaming (lower price, lower quality and worse experience).

 

[Shifty Geezer]

What I'm saying is that if the camera was tracking at 60FPS or higher it would open the door for more Kinect games because devs could make games that use twitch gameplay.

There was/is significant latency with Kinect regardless of framerate. I suppose if some of that is requiring a history of motion vectors, 60 fps could reduce latency somewhat, but motion gaming is rarely about twitch gaming. It's more naturally suited to timing, movement, and body control. For twitch, nothing beats buttons or something placed in the hands where our motion control abilities are largely focussed.

 

[onQ]

How it's better for gaming than Kinect 2? It has less accuracy than Kinect 1. Higher FPS would be important if it had equal or better accuracy than Kinect 1/2. Also, Kinect 2 is 60 fps capable.

Spoiler

http://forum.beyond3d.com/showpost.php?p=1849343&postcount=272

PS4 camera supports three modes:

1)1280x800p @60fps
2)640x400p @120fps
3)320x192p @240fps

Its quality will be lower at higher FPS, since lower resolution means lower accuracy (you have less pixels to understand object shape/movement from, correctly) and higher FPS needs more compute resources. Also PS4 camera may have some problems with dark-skinned users, ambient lighting and different distances and we know nothing about latency.

So I think it's only cheaper than Kinect 1/2, not better, for motion gaming (lower price, lower quality and worse experience).

Please show me where Kinect 2 is 60FPS? because the Prime Sense sensor was also 60FPS but Kinect 1 was 30FPS.

Lower price doesn't = lower quality and worse experience. the fact is there is more than one way to skin a cat & Sony decided to use Stereo Vision & compute for their depth sensing which is cheaper because it use 2 RGB cameras instead of a Time of Flight camera.

There was/is significant latency with Kinect regardless of framerate. I suppose if some of that is requiring a history of motion vectors, 60 fps could reduce latency somewhat, but motion gaming is rarely about twitch gaming. It's more naturally suited to timing, movement, and body control. For twitch, nothing beats buttons or something placed in the hands where our motion control abilities are largely focussed.

Would you want to control a 60FPS game using 1 frame of input data for every 2 frames of gameplay? Also would you want to be limited to only making 30FPS games when control is important but you're not able to buffer the input data because the game needs the input data as it come?

 

[function]

The performance parameters for the depth sensor state that Kinect 2 can go up to 60 fps.

Unfortunately, MS must have decided that you only need 30 fps for waving at the tv, trying to grab movies and eat them as you huff "Bing pizza" at Kinect.

The extra 16 ms for 30 fps will only be part of the pipeline that ends up with things on your screen. Hopefully MS will enable a 60 fps output mode though, or allow 60 fps from the camera to combine with 30 fps "genuine" depth tracking for effective 60 fps.

I feel for sorry for the engineers that developed Kinect 2. There's no doubt that Sony's camera solution is vastly inferior to Microsoft's, but if Sony's ends up having better games then that hardly matters, does it?

 

[onQ]

The performance parameters for the depth sensor state that Kinect 2 can go up to 60 fps.

Unfortunately, MS must have decided that you only need 30 fps for waving at the tv, trying to grab movies and eat them as you huff "Bing pizza" at Kinect.

The extra 16 ms for 30 fps will only be part of the pipeline that ends up with things on your screen. Hopefully MS will enable a 60 fps output mode though, or allow 60 fps from the camera to combine with 30 fps "genuine" depth tracking for effective 60 fps.

I feel for sorry for the engineers that developed Kinect 2. There's no doubt that Sony's camera solution is vastly inferior to Microsoft's, but if Sony's ends up having better games then that hardly matters, does it?

You say it's inferior yet the end result of the control input is 60FPS , 21 joint skeleton tracking. plus it can be used with the PlayStation Move.

Notice how people have to adjust to the slower movements when controlling a avatar with Kinect? it's something that you naturally do that's what I'm talking about when I say that Kinect's 30FPS tracking weighs down the gameplay when you are controlling a avatar. Notice the part in the DepthSense demo when the guy was doing the sword fighting with the PlayStation Move how he was moving more freely? Even if you watch yourself on a slow webcam you will adjust your movements to match it it's just something that you do but if you have a PlayStation Eye or fast webcam take it to 60 FPS or 120FPS you feel free and just be moving fast it's weird but it's true.

 

[Shifty Geezer]

Would you want to control a 60FPS game using 1 frame of input data for every 2 frames of gameplay? Also would you want to be limited to only making 30FPS games when control is important but you're not able to buffer the input data because the game needs the input data as it come?

Perhaps we're butting heads against choice of words again? Whether Kinect is 30 fps or 60 fps, I expect the game design choices to remain the same. I don't see anything one could do at 60 fps based on arm waving head wobbling that wouldn't also work at 30 fps, only less well. Things like sword fighting are more timing based than twitch based, because you see the opponent's movements well ahead of when they strike and intercept. You don't want too much lag, of course, but that sort of game style won't be enabled at 60 fps and impossible at 30 fps.

Kinect's 30FPS tracking weighs down the gameplay

Yes, it makes it more sluggish I'll agree, although it's not proven that running the IR feed on Kinect at 60 fps will result in much smoother tracking. There's a lot of processing overhead. Switching to 60fps may just mean an 17-33 ms saving latency on each frame delay. That's also true of SoftKinect. Latency will be more a matter of how much needs to be buffered.

Basically, the 30/60 fps input argument is the same as the 30/60 fps output argument. Higher framerates are smoother, but don't generally open up new gameplay types. Lower latency could make some game styles that were tried unsuccessfully on other systems more palatable for consumers though.

I don't see PS4 running 60 fps opening up more game styles to motion gaming. It'll be the same set of game styles, only a bit smoother if the tracking tech is as good as it promises (which is rarely the case).

 

[mosen]

Please show me where Kinect 2 is 60FPS? because the Prime Sense sensor was also 60FPS but Kinect 1 was 30FPS.

Lower price doesn't = lower quality and worse experience. the fact is there is more than one way to skin a cat & Sony decided to use Stereo Vision & compute for their depth sensing which is cheaper because it use 2 RGB cameras instead of a Time of Flight camera.

Kinect 1 was also 60fps capable on PC.

I didn't say lower price leads to lower quality, but stereo vision cameras have lower quality than ToF cameras (in this case it has lower quality than even Kinect 1). They also need more compute resources to operate and you can't compare their solution with Kinect 1/2 since we know nothing about delay/latency of their solution.

Kinect 2 end to end latency is 34ms. Less than 14 ms to sends a packet of data (the color image, depth data, and sound) plus 20ms latency to software. Do you have similar numbers for PS4 camera?

You say it's inferior yet the end result of the control input is 60FPS , 21 joint skeleton tracking. plus it can be used with the PlayStation Move.

Notice how people have to adjust to the slower movements when controlling a avatar with Kinect? it's something that you naturally do that's what I'm talking about when I say that Kinect's 30FPS tracking weighs down the gameplay when you are controlling a avatar. Notice the part in the DepthSense demo when the guy was doing the sword fighting with the PlayStation Move how he was moving more freely? Even if you watch yourself on a slow webcam you will adjust your movements to match it it's just something that you do but if you have a PlayStation Eye or fast webcam take it to 60 FPS or 120FPS you feel free and just be moving fast it's weird but it's true.

This video shows nothing, she tends to wave her hands slowly to show smooth tracking capability of the new Kinect which is what they want to do. I played many Kinect 1 games and I didn't need to adjust my movement with avatars.

 

[Shifty Geezer]

They also need more compute resources to operate and you can't compare their solution with Kinect 1/2 since we know nothing about delay/latency of their solution.

We have a video showing seemingly low latency when you watch the guy with the sword kicking blocks.

This video shows nothing.

It shows a very obvious lag between her movements and the avatar. It's most perceptible when she waves her arms up and down while walking side to side. Her arm is up when the avatar's arm is pretty much down.