Blazing Fast NVMEs and Direct Storage API for PCs *spawn*

[DavidGraham]

- Epic did say that PS5s SSD solution is faster than what you can find on current top end PCs (true)

Not exactly, even 6 months ago there were solutions that are way faster than PS5. Gigabyte launched an SSD that is capable of reaching 15GB/s, it's called AORUS Gen4 AIC SSD 8 TB.
https://www.amazon.com/Gigabyte-Performance-Advanced-Solution-GP-ASACNE6800TTTDA/dp/B081BSF14V

Other less exotic solutions include the Viper VP4100, Corsair MP600, and AORUS NVMe Gen 4, all of which already provide 5GB/s speeds.

 

[Arwin]

But that is not the whole story. He mentioned that the way the SSD is integrated into the system is beyond what Windows offers currently. And I do believe that the optimized file system, the additional interrupts and the integration into the graphics pipeline all combine to make it very effective.

That is not to say PC won’t catch up of course. Eventually they always do and they certainly likely will have by the time the new consoles are out, perhaps not quite as efficient but then through brute force. Still it will be interesting to see when the PC can match the OS5s extremely low latency from SSD to graphics pipeline.

 

[Deleted member 11852]

Not exactly, even 6 months ago there were solutions that are way faster than PS5. Gigabyte launched an SSD that is capable of reaching 15GB/s, it's called AORUS Gen4 AIC SSD 8 TB.
https://www.amazon.com/Gigabyte-Performance-Advanced-Solution-GP-ASACNE6800TTTDA/dp/B081BSF14V

These parts do have theoretical speeds rivalling/exceeding PS5, what these parts do not solve is the problem of pulling data off the SSD into RAM, for CPU unpacking/decompression and for anything for the GPU, shoving that all over another bus before it's ready to use.

 

[DavidGraham]

what these parts do not solve is the problem of pulling data off the SSD into RAM, for CPU unpacking/decompression and for anything for the GPU, shoving that all over another bus before it's ready to use.

It's also worth noting that PS5's solution includes optimised OS access and intrinsic compression, so it's performance goes beyond the basic 5 GB/s rate

Shouldn't DirectStorage be the PC answer to all of this?

We also know that PC's don't use Kraken compression (at least not for everything), and there seems to be some debate as to whether all data on a PC drive would need to be decompressed in software before use or not

Several games don't require compression at all on PCs, a famous case is Titanfall, it came with 30GB of uncompressed sound data because the developer wanted the game to run on low end CPUs.

Were they the same settings?

Yes they were, it's pointless for the engineer to run the demo with reduced detail on a laptop and compare that directly to PS5. He did it to prove a point, that current gen runs the new features. Epic themselves stated the demo runs with good performance on current NVMe drives, they didn't mention once that the demo got downgraded. In fact we now have double confirmation that current PCs can run the demo.


First: RTX 2070 Super and fast NVMe drive runs the demo with pretty good performance.
Second: RTX 2080 laptop and Smasung Evo 970 ran the demo with 40fps.

So that's two similar information bits in a row right now, with more likely incoming in the upcoming weeks

 

[Deleted member 11852]

Shouldn't DirectStorage be the PC answer to all of this?

For the last time, no. A software API does not negate all of the driver and device I/O that Windows does. The fact you keep suggesting this strongly suggests you need to spend a good hour reading this Microsoft documentation.

Once you realise how much work is actually involved in pulling one asset from a pack (or even an isolated file) in Windows, you'll understand why DirectStorage cannot undo all of that because fundamentally to Windows your PC is just a bunch of components that needs Windows, its filesystem and Windows kernel drivers to move small bits of data around in an incredibly administratively burdensome way. I'm talking thousands of lines off code and API calls just to open and find data in a file because to get to the data you need the device drivers for the storage system, all of the interconnect buses between the CPU and RAM and the storage and Windows ranges each movement and handover.

The only way Microsoft can fix this in software is to toss pretty much most of the Windows code in the bin, along with the filesystem, and start over. This will break everything. It's a non starter.

This is why no amount of money can buy an SSD RAID array that eliminates all load times in Windows games. There are too many individual pieces of hardware involved, all need to talk to each other through a different bunch of device drivers that themselves cannot communicate directly with each other. And this chain doesn't operate at the same speed so bottlenecks are common, and throughout the process Windows manages the data in ways to keep it secure from any rogue code that might be running.

 

[chris1515]

https://twitter.com/x/status/1262581494968508416

Unbuffered I/O on PC after it must need a very good CPU

https://twitter.com/x/status/1248426555094609924

https://social.technet.microsoft.co...obocopy-unbuffered-io?forum=win10itprogeneral

Some explanation about unbuffered I/O

 

[zupallinere]

Once you realise how much work is actually involved in pulling one asset from a pack (or even an isolated file) in Windows, you'll understand why DirectStorage cannot undo all of that because fundamentally to Windows your PC is just a bunch of components that needs Windows, its filesystem and Windows kernel drivers to move small bits of data around in an incredibly administratively burdensome way. I'm talking thousands of lines off code and API calls just to open and find data in a file because to get to the data you need the device drivers for the storage system, all of the interconnect buses between the CPU and RAM and the storage and Windows ranges each movement and handover.

With Windows it's Bueracracies All The Way Down.

 

[Shifty Geezer]

Unbuffered I/O on PC after it must need a very good CPU

Is this open to games?

 

[Scott_Arm]

@DSoup Microsoft is claiming that DirectStorage will reduce 5 CPU cores worth of processing (3 hardware decompression, 2 improvements to I/O) down to one tenth of one core. They haven't explained how the new I/O system works, but it sounds like it basically bypasses your typical filesystem access and all of that Windows OS bureaucracy.

The final component in the triumvirate is an extension to DirectX - DirectStorage - a necessary upgrade bearing in mind that existing file I/O protocols are knocking on for 30 years old, and in their current form would require two Zen CPU cores simply to cover the overhead, which DirectStorage reduces to just one tenth of single core.

"Plus it has other benefits," enthuses Andrew Goossen. "It's less latent and it saves a ton of CPU. With the best competitive solution, we found doing decompression software to match the SSD rate would have consumed three Zen 2 CPU cores. When you add in the IO CPU overhead, that's another two cores. So the resulting workload would have completely consumed five Zen 2 CPU cores when now it only takes a tenth of a CPU core.

https://www.eurogamer.net/articles/digitalfoundry-2020-inside-xbox-series-x-full-specs

 

[Jay]

https://twitter.com/x/status/1262581494968508416

Unbuffered I/O on PC after it must need a very good CPU

https://twitter.com/x/status/1248426555094609924

https://social.technet.microsoft.co...obocopy-unbuffered-io?forum=win10itprogeneral

Some explanation about unbuffered I/O

Sorry not read it, but could some of the issues raised be what direct storage will potentially address?

 

[DavidGraham]

For the last time, no. A software API does not negate all of the driver and device I/O that Windows does. The fact you keep suggesting this strongly suggests you need to spend a good hour reading this Microsoft documentation.

But that's precisely what Microsoft is promising .. I don't see how can you be so decisive with this no, like that. DirectStorage will be basically a lower level API just like DX12/Vulkan, bypassing a lot of OS/driver overhead.

 

[Deleted member 11852]

@DSoup Microsoft is claiming that DirectStorage will reduce 5 CPU cores worth of processing (3 hardware decompression, 2 improvements to I/O) down to one tenth of one core. They haven't explained how the new I/O system works, but it sounds like it basically bypasses your typical filesystem access and all of that Windows OS bureaucracy.

https://www.eurogamer.net/articles/digitalfoundry-2020-inside-xbox-series-x-full-specs

On Xbox Series X, hence the references to Zen cores as the metric to demonstrate performance improvements. What Microsoft's Andrew Goossen is demonstrating here is how much the Velocity Architecture I/O would have saturated the software stack if Microsoft had not changed it and introduced DirectStorage.

But that's precisely what Microsoft is promising .. I don't see how can you be so decisive with this no, like that. DirectStorage will be basically a lower level API just like DX12/Vulkan, bypassing a lot of OS/driver overhead.

Microsoft are "precisely" "promising" nothing with regard to the Windows version of DirectStorage, other than "we plan to bring to Windows". A Windows PC is nothing like an Xbox. Microsoft know exactly what hardware and specification of hardware is in every Xbox meaning they are not reliant on a bunch of device drivers to manage the I/O handover. PCs will still have two RAM pools and much longer chain of interconnect buses to get data from the SSD to the controller to the bus to main RAM for the CPU and across the PCI to GDDR.

You need to stop believing that this software unicorn can re-architect your hardware, because it cannot. Every single desktop OS works like this for a reason, and can only work like this. The only way to change it would be for an OS to mandate very specific hardware and remove the need for a bunch of different device drivers. Have a look at Device Manager at all the different device drivers that make your PC work. Learn how your PC works, then you'll understand.

Don't forget about the mythical GameCore integration. Wouldn't surprise me that parts of DirectX 12 Ultimate have yet to be fully exposed. If what has been hinted at comes to fruition it might bring more console-like integration to PC games.

No, for the same reason.

 

[Scott_Arm]

@DSoup why would Microsoft bring directstorage to pc if it didn’t address any issues? I’m expecting it’ll work around some file system performance issues, even though it can’t address the issues with the hardware architecture.

 

[Unknown Soldier]

@DSoup why would Microsoft bring directstorage to pc if it didn’t address any issues? I’m expecting it’ll work around some file system performance issues, even though it can’t address the issues with the hardware architecture.

DSoup said as much

A Windows PC is nothing like an Xbox. Microsoft know exactly what hardware and specification of hardware is in every Xbox meaning they are not reliant on a bunch of device drivers to manage the I/O handover. PCs will still have two RAM pools and much longer chain of interconnect buses to get data from the SSD to the controller to the bus to main RAM for the CPU and across the PCI to GDDR.

You need to stop believing that this software unicorn can re-architect your hardware, because it cannot.

 

[Unknown Soldier]

That's not answering his question at all.

But DSoup won't know what MS know. If MS feel there's a benefit, even if it's 10%, MS will add DirectStorage to DirectX Ultimate for PC. Any benefit is good. Will it reach the XBSX level benefit? I highly doubt but again any benefit would be good.

 

[DavidGraham]

PCs will still have two RAM pools and much longer chain of interconnect buses to get data from the SSD to the controller to the bus to main RAM for the CPU and across the PCI to GDDR.

We know this chain will always be here, in fact it is one of PC stregnths (the double RAM pool), to allow for more powerful hardware without compromise. This isn't the crux of the problem at all by the way.

You previously stated that the main problem is the file system and the OS/APIs, now when Microsoft is stating they can alleviate alot of that you are just going to shrug it off?

The only way to change it would be for an OS to mandate very specific hardware and remove the need for a bunch of different device drivers.

Whose to say that DirectStorage will not require compatible SSDs just like DX requires compatible video and audio cards?

You need to stop believing that this software unicorn can re-architect your hardware, ..
Learn how your PC works, then you'll understand.

And you need to stop this authoritative confirmatory tone about something you know basically nothing about.

 

[Deleted member 11852]

We know this chain will always be here, in fact it is one of PC stregnths (the double RAM pool), to allow for more powerful hardware without compromise. This isn't the crux of the problem at all by the way.

No, two pools, or rather the existence of a further bus and device drive (PCI, GPU) exasperates the problem.

You previously stated that the main problem is the file system and the OS/APIs, now when Microsoft is stating they can alleviate alot of that you are just going to shrug it off?

Microsoft's comments are about Xbox, not PC. They have said and promised nothing regarding PC other than planning to bring DirectStorage to Windows. If you think this, please cite it or stop claiming this.

Whose to say that DirectStorage will not require compatible SSDs just like DX requires compatible video and audio cards?

How does compatible SSD drive help? How does that undue the chain of SSD -> controller -> southbridge -> RAM -> Northbridge -> GDDR -> GPU. How does that change anything? Your comments only demonstrate you don't know how your PC works, or how for data to transfer from SSD to RAM you need to involve the device driver for the SSD, the overarching interface driver, and the interconnect bus driver. And that's just for starters. What different does one SSD or another make?

Please. Read the links I posted before. It explains how Windows is architecturally designed.

And you need to stop this authoritative confirmatory tone about something you know basically nothing about.

I've written Windows device drivers. You don't even appear to understand what they are.

 

[DavidGraham]

Microsoft's comments are about Xbox, not PC. They have said and promised nothing regarding PC other than planning to bring DirectStorage to Windows. If you think this, please cite it or stop claiming this.

Their message to DigitalFoundry certainly came across as planning to bring the same kind of improvements to PCs, but even if they are not the same, there stands to be some definite improvements otherwise why would they even bother?

How does compatible SSD drive help? How does that undue the chain of SSD -> controller -> southbridge -> RAM -> Northbridge -> GDDR -> GPU. How does that change anything? Your comments only demonstrate you don't know how your PC works, or how for data to transfer from SSD to RAM you need to involve the device driver for the SSD, the overarching interface driver, and the interconnect bus driver. And that's just for starters. What different does one SSD or another make?

There are two problems to the current paradigm that you gracefully explained:
1-Too many buses
2-Cumbersome File Systems/APIs/OS
Are you saying there is no way in hell any of these problems would be alleviated by a new API?

I've written Windows device drivers. You don't even appear to understand what they are.

Pulling the old "I've written drivers and you didn't" trick seems both childish and immature at this point, especially in the face of the massive shift in storage paradigms, I could throw this right back at ya by asking how many APIs have you written? For which the answer would be?

 

[Deleted member 11852]

Their message to DigitalFoundry certainly came across as planning to bring the same kind of improvements to PCs, but even if they are not the same, there stands to be some definite improvements otherwise why would they even bother?

I didn't read it as Microsoft promising improvements. They just they're planning to bring it Windows. Why would they bring an API that des nothing to Windows? Because Microsoft need API parity for games that work on both Xbox and Windows. Even if DirectStorage effectively does nothing - it needs to be there for game code using it on Xbox not to break when running on PC.

There are two problems to the current paradigm that you gracefully explained:
1-Too many buses
2-Cumbersome File Systems/APIs/OS
Are you saying there is no way in hell any of these problems would be alleviated by a new API?

A new API does not make all the buses in your PC disappear. This is how your hardware is all connected to each other. This won't change. DirectStorage is not a new filesystem. Could it bring some improvements? Sure but given the current Windows driver architecture, I'm not seeing how unless Microsoft want to fundamentally change the Windows model. They never have, we've never had a revolution in architecture, only evolution.

Pulling the old "I've written drivers and you didn't" trick seems both childish and immature at this point, especially in the face of the massive shift in storage paradigms, I could throw this right back at ya by asking how many APIs have you written? For which the answer would be?

Bullshit. You accused me of using "this authoritative confirmatory tone about something you know basically nothing about.". If you attack someone's perceived knowledge don't be surprised them they cite their experience or credentials. You invited that response.

As for your query on APIs, I've also written a lot of APIs for bespoke hardware and shared software libraries, far too many to remember them all. I bet most experienced programmers have. Anybody who creates something for use by others has written an API, that is a vast, vast amount of coding effort in the world. But I don't see the relevance?

 

[DavidGraham]

Because Microsoft need API parity for games that work on both Xbox and Windows. Even if DirectStorage effectively does nothing - it needs to be there for game code using it on Xbox not to break when running on PC.

Highly unconvincing to be frank, If DirectStorage is directed mainly at Series X, then there would be no need to bring it to PCs, console APIs don't get transferred to PCs to maintain feature parity, there is no need to (especially with something completely busted as you describe it), games on PC work with the already established systems. You don't
See Sony bringing their APIs on PC.

A new API does not make all the buses in your PC disappear.

But it can enable various workarounds. Give rise to new devices with new capabilities.

Could it bring some improvements? Sure but given the current Windows driver architecture, I'm not seeing how unless Microsoft want to fundamentally change the Windows model.

You don't know that, that was my original remark in the first place, you don't know what Microsoft's plan for DirectStorage is, they might be planning for a revolutionary change (just like Series X), they might not be, we still don't know, yet if we follow your line of thinking we would have already declared their effort worthless.

If you attack someone's perceived knowledge don't be surprised them they cite their experience or credentials.

It's very good that you know about this rule, sadly you don't seem to follow it at all, you invited this response and others when you used a confirmatory tone about something you know nothing about "DirectStorage", and when you constantly implied my (and others) lack of knowledge about basic filesystem/OS principles. No one doubted your knowledge and experience in the first place, yet you proceeded on your own to hand everyone around you (You are an ignorant) card, and that behavior is simply unacceptable and quite frankly rude.