Support for Machine Learning (ML) on PS5 and Series X?

[patsu]

Tempest looks like the worst place to do it, at least for any real-time image processing for a game. PS5 gpu should support like 20.0 TFLOPS with FP16. Tempest is maybe 0.2 - 0.4 TFLOPS with FP16. An RTX2060 is 52 TFLOPS with FP16 on tensor cores, for comparison.

There are many different Machine Learning applications. You’ll have to pick your solutions wisely.

If Tempest can fix < 5ms latency audio issues, it may be able to tackle issues with similar real-time needs.

It should be in the back pocket of developers for on-device inference. They are not all heavy weight too. It’d be overkill.

When looking for a solution, I’d pick the lowest cost solution, not the highest spec’ed ones first.

I don't suppose they're working with Khronos for a VulkanML off-shoot of sorts.

That would be my starting point !

AFAIK, RDNA2 extended support for ML means low precision types and dot product instructions executed on SFUs. I think Navi 12 already has them, which is in (future) products exclusive for Apple?
Assuming dot product SFU takes much less die area than NVs tensors with matrix ops, it's quite likely Sony would have adopted them and there is no hardware difference between consoles.

*nod* *nod*

Apple has NeuralEngine. So those concepts are built into their API today. GPU vendors are at the forefront of AI. Would be surprised if they are behind.

 

[Jay]

Assuming dot product SFU takes much less die area than NVs tensors with matrix ops, it's quite likely Sony would have adopted them and there is no hardware difference between consoles.

Unlike other features my starting point is that PS5 doesn't have the reduced precision formats.

The question is, by default what advantages do they give. Every feature may not cost much die space but does cost to have it included unless it's part of the base architecture.
MS specified the reason they included it was due to azure.
Would they have added it otherwise?
Possibly, but taken at face value no.

DF, although this isn't where they mention it was due to azure, it was someplace else.

We knew that many inference algorithms need only 8-bit and 4-bit integer positions for weights and the math operations involving those weights comprise the bulk of the performance overhead for those algorithms," says Andrew Goossen. "So we added special hardware support for this specific scenario. The result is that Series X offers 49 TOPS for 8-bit integer operations and 97 TOPS for 4-bit integer operations. Note that the weights are integers, so those are TOPS and not TFLOPs. The net result is that Series X offers unparalleled intelligence for machine learning.

 

[patsu]

There are good reasons why Sony should keep those existing GPU features.
Not just "Azure needs them".

Software and algorithm advancements are as significant as hardware ones.
ML does wonders in image processing, signal processing, and many other relevant areas.

It's not end of the world, but when in-play, those basic data types and primitive operations get used *a lot*.
Quantized/reduced precision models are common, especially in small devices; they save memory footprint + improve performances.

(We have to double check whether those quantized models are indeed used in gaming and AV related domains. If they are rejected for some good reasons, let's hear them)

 

[Jay]

There are good reasons why Sony should keep those existing GPU features.
Not just "Azure needs them".

That's my point.
Your talking about keeping it.
I'm talking about adding it.
It depends if its part of the base IP or an additional feature that can be bought.

Going by MS statement sounds like it's not part of the base IP. Obviously taking it at face value.
Is it useful, sure. But is it worth paying additional for?
MS said they added it.

 

[patsu]

For example ...

Yes (!) if Sony is remotely interested in game streaming (Didn't they partner with Microsoft in this area ?)



Base IP or not, the design is done and proven. Feature is well supported and used everywhere. Take it ~

 

[Cyan]

The toolkits for *on-device* ML inference should be light weight and low latency because they run on cell phones.

AMD and Sony should both already have optimized math libraries, but these may lack ML-specific hardware support (e.g. INT4 ?).

It's the end-to-end ML toolsets that's harder to come together (on AMD platforms). Nvidia has it all covered. I haven't personally tried AMD ROCm.

hey, long time no see! There is no news in regards to a solution similar to nVidia's DLSS, not even in the leaked AMD docuemnts, which means that the gap between consoles and PCs is going to increase even more.

 

[eastmen]

hey, long time no see! There is no news in regards to a solution similar to nVidia's DLSS, not even in the leaked AMD docuemnts, which means that the gap between consoles and PCs is going to increase even more.

isn't most of dlss done in cloud compute ? That's why we have to wait for nvidia to add game support to it? What would stop microsoft from putting up hundreds of series x's in azure and allowing devs to access it to .

Also RDNA 1 has Radeon Image sharpening and they are also working on Fidelity FX
https://www.digitaltrends.com/computing/amd-radeon-image-sharpening-dlss-ray-tracing-e3-2019/

Digitaltrends seemed impressed with it back then and I am sure RDNA 2 will also have it , if not an enhanced version
"MD had both solutions running Battlefield V side-by-side — the Nvidia system at 4K with ray tracing and DLSS on, and the AMD system with Radeon Image Sharpening in 1440p. The result was very similar, with the AMD system looking close to 4K thanks to the added sharpness of RIS. And, because it was still technically playing at 1440p, Battlefield V hit framerates upward of 90 FPS."
It apparently just works on dx9 , 12 and vulkan without needing special support like dlss.

The improved version will work with all cards

 

[Cyan]

isn't most of dlss done in cloud compute ? That's why we have to wait for nvidia to add game support to it? What would stop microsoft from putting up hundreds of series x's in azure and allowing devs to access it to .

Also RDNA 1 has Radeon Image sharpening and they are also working on Fidelity FX
https://www.digitaltrends.com/computing/amd-radeon-image-sharpening-dlss-ray-tracing-e3-2019/

Digitaltrends seemed impressed with it back then and I am sure RDNA 2 will also have it , if not an enhanced version
"MD had both solutions running Battlefield V side-by-side — the Nvidia system at 4K with ray tracing and DLSS on, and the AMD system with Radeon Image Sharpening in 1440p. The result was very similar, with the AMD system looking close to 4K thanks to the added sharpness of RIS. And, because it was still technically playing at 1440p, Battlefield V hit framerates upward of 90 FPS."
It apparently just works on dx9 , 12 and vulkan without needing special support like dlss.

The improved version will work with all cards

now that you mention it, I think Resident Evil 3 Remake uses FidelityFX upscaling, even on nVidia GPUs, but how well it works... I dunno. I have the demo, and set the game at 1440p. Gotta try reducing the resolution to 720p and see what happens.

 

[PSman1700]

The tensor cores are supposed to be helping out regarding dlss.

 

[RobertR1]

You might want to come upto speed on DLSS 2.0 @eastmen

 

[JoeJ]

Unlike other features my starting point is that PS5 doesn't have the reduced precision formats.

The question is, by default what advantages do they give.

Yeah, you may be right. I don't know what the area costs really are. Even a 1% could be too much if somebody is not convinced about ML in games yet.
Offering hardware acceleration before it has been shown a feature is really needed is a really paradox situation. To me, upscaling alone is not enough to proof demand. We'll likely see more applications, but not sure if it's ready during next gen.

 

[Dictator]

isn't most of dlss done in cloud compute ? That's why we have to wait for nvidia to add game support to it? What would stop microsoft from putting up hundreds of series x's in azure and allowing devs to access it to .

Also RDNA 1 has Radeon Image sharpening and they are also working on Fidelity FX
https://www.digitaltrends.com/computing/amd-radeon-image-sharpening-dlss-ray-tracing-e3-2019/

Digitaltrends seemed impressed with it back then and I am sure RDNA 2 will also have it , if not an enhanced version
"MD had both solutions running Battlefield V side-by-side — the Nvidia system at 4K with ray tracing and DLSS on, and the AMD system with Radeon Image Sharpening in 1440p. The result was very similar, with the AMD system looking close to 4K thanks to the added sharpness of RIS. And, because it was still technically playing at 1440p, Battlefield V hit framerates upward of 90 FPS."
It apparently just works on dx9 , 12 and vulkan without needing special support like dlss.

The improved version will work with all cards

Radeon Image Sharpening or Contrast adaptive sharpening is nothing like image reconstruction done by DLSS or any other reconstruction scheme. It is just a post process to increase contrast locally, it does not make new pixels, fill in lines, or do anti-aliasing. It actually increases aliasing.
DLSS is Machine Learning work done locally on the GPU - looking at a frame and previous frames to reconstruct a higher resolution frame. The information decididing how that machine learning work is done on the GPU, as in the Neural Network Weights, are done by NV separately on super computers.

 

[iroboto]

Offering hardware acceleration before it has been shown a feature is really needed is a really paradox situation.

Is offering int4/8 really hardware acceleration? Or just effective use of silicon for lower precision?

 

[JoeJ]

Is offering int4/8 really hardware acceleration? Or just effective use of silicon for lower precision?

No matter how we call it, the question is what's the cost and benefit.
We really should discuss application of ML (and low precision types or dot product instructions) in general. I'd like to know what visions and ideas people have, beyond upscaling. But not even NV can show something really related to games yet.

 

[eastmen]

You might want to come upto speed on DLSS 2.0 @eastmen

I've read the DF control article. Did they remove the requirement for them to have first ran it through their machine learning hundreds of thousands of times before allowing the local gpu to do DLSS ? Cause I haven't read anyone

Radeon Image Sharpening or Contrast adaptive sharpening is nothing like image reconstruction done by DLSS or any other reconstruction scheme. It is just a post process to increase contrast locally, it does not make new pixels, fill in lines, or do anti-aliasing. It actually increases aliasing.
DLSS is Machine Learning work done locally on the GPU - looking at a frame and previous frames to reconstruct a higher resolution frame. The information decididing how that machine learning work is done on the GPU, as in the Neural Network Weights, are done by NV separately on super computers.

Your saying two different things here. If Nvidia is using the cloud for MS learning first and having to run the game thousands of times on the cloud servers before enabling it locally . There is a step then that can't be done on just tensor cores, otherwise you'd be able to just run it on any game.

Now I haven't looked into RIS as much as I should but again I don't see why sony or ms can't use the cloud to do a similar step but use shader cores locally. I guess we will have to wait for more details about RDNA 2 and what its capable of

 

[SimBy]

Radeon Image Sharpening or Contrast adaptive sharpening is nothing like image reconstruction done by DLSS or any other reconstruction scheme. It is just a post process to increase contrast locally, it does not make new pixels, fill in lines, or do anti-aliasing. It actually increases aliasing.
DLSS is Machine Learning work done locally on the GPU - looking at a frame and previous frames to reconstruct a higher resolution frame. The information decididing how that machine learning work is done on the GPU, as in the Neural Network Weights, are done by NV separately on super computers.

What matters is end result not how it's done. There's no denying RIS was at least as good if not better in many ways than DLSS 1.x And obviously AMD will (have to) come up with something even better.

 

[iroboto]

I've read the DF control article. Did they remove the requirement for them to have first ran it through their machine learning hundreds of thousands of times before allowing the local gpu to do DLSS ? Cause I haven't read anyone

Your saying two different things here. If Nvidia is using the cloud for MS learning first and having to run the game thousands of times on the cloud servers before enabling it locally . There is a step then that can't be done on just tensor cores, otherwise you'd be able to just run it on any game.

Now I haven't looked into RIS as much as I should but again I don't see why sony or ms can't use the cloud to do a similar step but use shader cores locally. I guess we will have to wait for more details about RDNA 2 and what its capable of

I think it would be helpful to read some source materials on how it would work.
But the gist of it is that DLSS requires a trained model to run.
The GPU is provided a trained model, in a format of a file, and the game uses that file to perform DLSS through it's hardware. In this case tensor cores.
This is all done locally, there is no cloud required here.

To build the trained model, requires training in the cloud, this is a completely different process. This is where they take source images and target images and they teach a Deep Learning algorithm to take X to become Y. That training takes a lot of time to do, so they do it with tons of computational power.

The algorithm can come in 2 formats: the first format is specific to a title, meaning you would need to train a new model for every single title, the second format is generic to all titles, meaning you can use a single model to infer for any type of game.

DLSS 2.0 is based on the latter, but that does not mean every game can implement it. The reason is because you must still meet the requirements for the input data, in this case, the algorithm only knows how to take aliased raw data at specific resolutions without reconstruction, to turn it into a whatever setting you want it to get to. Most _OLDER_ titles were built like this before TAA became the standard, so you must develop a new render path to (a) create the raw aliased frames fro DLSS and then you must use the DLSS file in that render path to output DLSS.

Then you can support DLSS. And if you don't get the results you want, then you can build a specific model using the DLSS2.0 model as a base and then adding some additional layers on top specific to the title and re-train adding pictures from the game to get better performance.

 

[eastmen]

I think it would be helpful to read some source materials on how it would work.
But the gist of it is that DLSS requires a trained model to run.
The GPU is provided a trained model, in a format of a file, and the game uses that file to perform DLSS through it's hardware. In this case tensor cores.
This is all done locally, there is no cloud required here.

so you need to the cloud to enable DLSS , it can't be done without first making that trained model.

To build the trained model, requires training in the cloud, this is a completely different process. This is where they take source images and target images and they teach a Deep Learning algorithm to take X to become Y. That training takes a lot of time to do, so they do it with tons of computational power.

right so they use a ton of cloud compute which is what I've siad

orithm can come in 2 formats: the first format is specific to a title, meaning you would need to train a new model for every single title, the second format is generic to all titles, meaning you can use a single model to infer for any type of game.

what are the advantages or disadvantages of both?

DLSS 2.0 is based on the latter, but that does not mean every game can implement it. The reason is because you must still meet the requirements for the input data, in this case, the algorithm only knows how to take aliased raw data at specific resolutions without reconstruction, to turn it into a whatever setting you want it to get to. Most _OLDER_ titles were built like this before TAA became the standard, so you must develop a new render path to (a) create the raw aliased frames fro DLSS and then you must use the DLSS file in that render path to output DLSS.

So you need a ton of cloud compute , you need to implement it on a case by case basis , it helps if you design the engine / game with DLSS in mind, It seems that MS has all of this and I am sure sony can get access to it also. So the only thing is that Nvidia is using tensor cores to do it and we don't know if there is a way for amd to do something similar with rdna 2 because we don't know much about it

Then you can support DLSS. And if you don't get the results you want, then you can build a specific model using the DLSS2.0 model as a base and then adding some additional layers on top specific to the title and re-train adding pictures from the game to get better performance.

Like I said it seems that right now its only certain games and even certain resolutions that are enabled for this.

 

[iroboto]

so you need to the cloud to enable DLSS , it can't be done without first making that trained model.

Correct it won't work without a trained model. You don't need the cloud necessarily, but its' just faster if you have that service.

what are the advantages or disadvantages of both?

You're trading off specific implementations in favour of trying to extract a little more graphical performance but at the cost of possibly overfitting/introducing more artifacts and also way more training time. So generally the first option is usually worse. We want AI to be generic, and to not be specific; we want it to adapt to many challenges it's never seen before and make the correct answer. But specific is easier to do than generic.

So you need a ton of cloud compute , you need to implement it on a case by case basis , it helps if you design the engine / game with DLSS in mind, It seems that MS has all of this and I am sure sony can get access to it also. So the only thing is that Nvidia is using tensor cores to do it and we don't know if there is a way for amd to do something similar with rdna 2 because we don't know much about it

Anyone can do this. If you don't have the cloud, you rent it. This is sort of why we put resources into the cloud to begin with, not everyone wants to pay for the infrastructure to train a model, so we share the resource and pay for compute time. In this case, Nvidia has already completed their generic DLSS model. It is now 2.0. They won't need to retrain a base ever again unless there is signficant changes they are making to their AI. They can just rely on transfer learning from here on out. Everyone else will need to do make their own generic AI. That would be costly, but a smart enough company could also pull it off and make competing models to Nvidia. And it can be a 3rd party company just looking at the math of licensing revenue.

Like I said it seems that right now its only certain games and even certain resolutions that are enabled for this.

Yea, there are hard restrictions on what we can do.

 

[Scott_Arm]

@eastmen No, you don't need the cloud to enable DLSS. It runs locally on your Nvidia gpu. The training is done by nvidia, but the DLSS program ships with the game.