Blazing Fast NVMEs and Direct Storage API for PCs *spawn*

[Remij]

I didn't read it as Microsoft promising improvements. They just they're planning to bring it Windows. Why would they bring an API that des nothing to Windows? Because Microsoft need API parity for games that work on both Xbox and Windows. Even if DirectStorage effectively does nothing - it needs to be there for game code using it on Xbox not to break when running on PC.

It'll be interesting to see your attitude when it's revealed just exactly what it does change and improve on PC... Will you graciously admit you were wrong?

"And here we have the newest member of the DirectX family, DirectStorage. It does this, and this, and helps with this, reduces this, and we plan on bringing it to PC as well... where it does nothing."

lol yea... ok

 

[Xbat]

It'll be interesting to see your attitude when it's revealed just exactly what it does change and improve on PC... Will you graciously admit you were wrong?

"And here we have the newest member of the DirectX family, DirectStorage. It does this, and this, and helps with this, reduces this, and we plan on bringing it to PC as well... where it does nothing."

lol yea... ok

Wait just a minute, Dsoup has given reasons why he doesn't think it's likely that direct storage for PC will be equivalent to Series X.

He never said it won't do anything just that he doesn't see it be the equivalent of what's doable on Series X.

 

[PSman1700]

if we follow your line of thinking we would have already declared their effort worthless.

Do't follow his thoughts I'm sure MS ain't stupid, they know better then 'concerned' people, xbox and pc are their platforms, every game will be on both, i don't see a huge disparity going to happen between xbox and pc. The nanite demo runs well on a 2080 max q laptop, supposedly better even, witout DS. PC users will be fine, like they allways heve been.

People will have to upgrade though, like every new console generation, bar 2013 ones then, where graphics didnt really get the jump we were used to.
Again, all will be fine, if your on PC, PS5 or XSX, a game centered around UE5 will run equally well on all platforms, between the consoles i doubt the difference will be that huge. If you got a 3080Ti pc with matching components to that ofc youre going to get the best experience overall, with that hefty price tag to it. It has been this way for every generation so far.

It'll be interesting to see your attitude when it's revealed just exactly what it does change and improve on PC... Will you graciously admit you were wrong?

Technically he already is, since UE5 demo supposedly does run on a (high end) laptop. I highly doubt they had it dailed back looking nothing like what was shown on PS5. To the contrary, i think it mst likely looked exactly the same (why bench otherwise), running at 40fps in editor mode.

 

[Deleted member 11852]

Highly unconvincing to be frank, If DirectStorage is directed mainly at Series X, then there would be no need to bring it to PCs, console APIs don't get transferred to PCs to maintain feature parity, there is no need to (especially with something completely busted as you describe it), games on PC work with the already established systems.

Microsoft couldn't support DirectStorage in UWP unless the API exists on all Microsoft platforms. The APIs have to common - source: Microsoft. Feel free to find Microsoft's own website unconvincing if you wish.

See Sony bringing their APIs on PC.

Because Sony are not promising a common platform for PlayStation and Windows. That is what Microsoft promised with UWP.

But it can enable various workarounds. Give rise to new devices with new capabilities.

Please be specific. How does a new API deliver better performance for all of the disparate components on your PC that mostly have different device drivers? We're in the console technology forum, not the baseless rumours thread. Please put out some specifics and explain what can be done differently and how it would help.

You don't know that, that was my original remark in the first place, you don't know what Microsoft's plan for DirectStorage is, they might be planning for a revolutionary change (just like Series X), they might not be, we still don't know, yet if we follow your line of thinking we would have already declared their effort worthless.

I've said in a few posts that some gains could be made, including here. But to make significant gains would require a huge change to the PC and/or Windows architecture, like absolutely massive. A bigger architectural change requiring revisiting the entire device driver model that would likely break all current device drivers. I don't think this is remotely feasible.

No one doubted your knowledge and experience in the first place, yet you proceeded on your own to hand everyone around you (You are an ignorant) card, and that behavior is simply unacceptable and quite frankly rude.

Yeah, you did. Just admit it. A couple of posts back you were asking if I'd written any APIs. To ask that after I'd already said I've written a device driver (a device driver without APIs?). Just drop it. I don't have the energy for this.

 

[Ronaldo8]

I didn't read it as Microsoft promising improvements. They just they're planning to bring it Windows. Why would they bring an API that des nothing to Windows? Because Microsoft need API parity for games that work on both Xbox and Windows. Even if DirectStorage effectively does nothing - it needs to be there for game code using it on Xbox not to break when running on PC.



A new API does not make all the buses in your PC disappear. This is how your hardware is all connected to each other. This won't change. DirectStorage is not a new filesystem. Could it bring some improvements? Sure but given the current Windows driver architecture, I'm not seeing how unless Microsoft want to fundamentally change the Windows model. They never have, we've never had a revolution in architecture, only evolution.



Bullshit. You accused me of using "this authoritative confirmatory tone about something you know basically nothing about.". If you attack someone's perceived knowledge don't be surprised them they cite their experience or credentials. You invited that response.

As for your query on APIs, I've also written a lot of APIs for bespoke hardware and shared software libraries, far too many to remember them all. I bet most experienced programmers have. Anybody who creates something for use by others has written an API, that is a vast, vast amount of coding effort in the world. But I don't see the relevance?

There is actually an Eurogamer interview with Jason Ronald where he gives the only known snippet of information about Directstorage. It seems that is indeed a new filesystem as anticipated by many commentators.

 

[Ronaldo8]

But I do agree that if it is indeed a new filesystem then I just don't see how it can be applied on current PC architectures. Seems counterintuitive.

 

[Deleted member 11852]

There is actually an Eurogamer interview with Jason Ronald where he gives the only known snippet of information about Directstorage. It seems that is indeed a new filesystem as anticipated by many commentators.

He says "a new filesystem API" not "a new filesystem". These are very different things but may be the cause of the confusion.

But again, he is specifically talking about Xbox where the read/write and file behaviour will be different (certainly more predictable) than a Windows PC. Even if Microsoft did want change the Series X filesystem, which isn't what he said, this wouldn't be particularly disruptive because it's an Xbox. Much Windows software - anything using deep/complex data structures, many utilities, recovery tools, local and cloud backup/sync tools - are predicated on the behaviour of NTFS.

If a new Windows filesystem were imminent, i.e. going toe delivered in the next three years, this would already be in dev channels for early testing. You don't just toss out a new filesystem, even Apple which has a smaller less complex and less diverse ecosystem, took three years to deploy APFS across its platforms and that was with all the abstraction groundwork (CoreStorage) completed. There is ReFS but that's been dormant for a while and was only slightly faster in some use cases than NTFS (and slower in others) because it's focussed on data resiliency, not performance.

 

[fehu]

If I'm not wrong, xboxes are already using a simplified version of ntfs

 

[Unknown Soldier]

There is actually an Eurogamer interview with Jason Ronald where he gives the only known snippet of information about Directstorage. It seems that is indeed a new filesystem as anticipated by many commentators.

Yep, talking about the XBox filesystem. As already said, even though Direct Storage API is apparently coming to PC, it shouldn't be the same or function the same as the Xbox version or it might not even have the same benefits as the Xbox system due to the different hardware architectures in the Xbox(closed) and PC(open). We will have no comparisons until the actual filesystem API hits PC and then people can compare, if that's possible.

 

[Shifty Geezer]

He says "a new filesystem API" not "a new filesystem". These are very different things but may be the cause of the confusion...

I agree completely with what you say, but, I think it not unreasonable to think he's using a sort of vernacular use like people here are. API is just the interface...why write a new one if it's not going to do anything different? So I can well believe in creating an new API, they also create a new end-to-end filesystem driven by that API as needed when saying it was a ground-up approach to fast IO, and it's just a quirk of language that he chose to talk about it in terms of an 'API'. I think 'API' is often used to mean 'low level system'.

 

[sir doris]

Not sure how related this could be, but all this talk over storage API's and file systems reminded me of the EPIC nvme ssd server Linus (LTT) made a few months back, where the performance was very poor compared to the theoretical performance. I can't remember exactly where why the performance was bad but it was something to do with either the CPU or North/ South bridge being overwhelmed by the speed of the ssds (IIRC nearly every channel it was fully populated with RAID nvme ssds). I'll have to re-watch the video when I have time. This is the first thing I thought of when Cerny announced the PS5 specs. Obviously the PS5 won't have anywhere near the throughput of that storage system, but that was purely used to store files as opposed to the PS5 which will be maxed out running games while the storage system is being utilised.



However the reason I posted this now is that, in Linus' recent video about Intel he mentions the issues they had with the EPIC server. He relates it, at least partially back to software, stating that he has never seen anything like that with Intel solutions. It was also software RAID too, but maybe it does give hope that streamlined software could boost the Windows IO performance?



LTT - SSD Storage

LTT - Why I still Love Intel

EDIT: Opps maybe this should be in the Next Gen Hardware thread

 

[chris1515]

I agree completely with what you say, but, I
think it not unreasonable to think he's using a sort of vernacular use like people here are. API is just the interface...why write a new one if it's not going to do anything different? So I can well believe in creating an new API, they also create a new end-to-end filesystem driven by that API as needed when saying it was a ground-up approach to fast IO, and it's just a quirk of language that he chose to talk about it in terms of an 'API'. I think 'API' is often used to mean 'low level system'.

Maybe they will do like Sony and create a file format only for games. The file archive is read only and only for the games data. Everything else use free BSD file system (OS, game data you need to write like save or world state...).

And like this, it has no effect out of games.

 

[Deleted member 11852]

I agree completely with what you say, but, I think it not unreasonable to think he's using a sort of vernacular use like people here are. API is just the interface...why write a new one if it's not going to do anything different?

Microsoft, Google and Apple change their APIs constantly, new functions get added, some functions get deprecated and sometimes APIs for function will change.

So I can well believe in creating an new API, they also create a new end-to-end filesystem driven by that API as needed when saying it was a ground-up approach to fast IO, and it's just a quirk of language that he chose to talk about it in terms of an 'API'. I think 'API' is often used to mean 'low level system'.

Microsoft have changed the API for NTFS far more than that they changed the actual filesystem, which has actually had relatively few changes given it's been around decades. If Microsoft have a new NTFS-replacement filesystem in the works, i.e. not ReFS, that is not already in dev channels yet (it isn't), then it is 3-4 years away at the earliest. You can't just flip the Windows filesystem on a dime.

This is why I don't think they're imminently talking about a new filesystem, or perhaps why they framed it was "we plan to bring it Windows" as in, in many years time.

 

[pjbliverpool]

I agree completely with what you say, but, I think it not unreasonable to think he's using a sort of vernacular use like people here are. API is just the interface...why write a new one if it's not going to do anything different? So I can well believe in creating an new API, they also create a new end-to-end filesystem driven by that API as needed when saying it was a ground-up approach to fast IO, and it's just a quirk of language that he chose to talk about it in terms of an 'API'. I think 'API' is often used to mean 'low level system'.

I don't have a strong view on this one way or the other since there's basically no public information available on it yet, and I don't understand it well enough regardless. But... from a non-technical point of view, this is clearly an issue. And thanks to the new consoles and massive speed increase in storage drives thanks to NVMe, it's becoming more and more apparent, very quickly.

The issue can't be left to fester indefinitely and now seems like the perfect time to try and address it. Microsoft are also the perfect company to address it, not only because of their position with Windows, but also from their position of wanting to unify the PC and Xbox platforms from a gaming perspective.

So the fact that Microsoft are coming up with something now, that on the Xbox at least is described as addressing these issues does seem promising. I get that it's hugely difficult and probably won't be done all at once for all the good reasons DSoup has laid out, but if they're not going to address, or at least start addressing the problem now, then when? There doesn't seem to be a better time.

Maybe they will do like Sony and create a file format only for games. The file archive is read only and only for the games data. Everything else use free BSD file system (OS, game data you need to write like save or world state...).

And like this, it has no effect out of games.

I'd wondered myself is something like this might be possible.

 

[Deleted member 11852]

So the fact that Microsoft are coming up with something now, that on the Xbox at least is described as addressing these issues does seem promising. I get that it's hugely difficult and probably won't be done all at once for all the good reasons DSoup has laid out, but if they're not going to address, or at least start addressing the problem now, then when? There doesn't seem to be a better time.

I 100% think Microsoft have plans to improve Windows I/O - as much as the PC and Windows architecture allows - but this is a mammoth task and there is literally zero evidence of it happening soon, which you would already expect to see signs of.

Apple spent a number of years paving the way to switch everybody using iOS and macOS to their new APFS filesytsem. This transition began back in 2014 with the introduction of CoreStorage API which allows rock solid logical addressing of storage devices - this is critical for filesystem conversion. After three years of of tweaking CoreStorage, Apple flipped the conversation switch for iOS in 2017 and for macOS on all-SSD partitions in 2018. Fusion-drives (hybrid SSD/HDD partitions) followed later. Effectively when you installed the latest OS, during an extended install/boot-up, your entire drive was converted to the new filesystem. It was very slick and also terrifying because.. data loss.

Compared to Windows, Apple's hardware ecosystem is very simple and Apple knew exactly hardware and drive controllers were in supported devices. Microsoft are not afforded this luxury and you can be sure that they want a seamless transition without people needing to backup, format and restore.

As for the actual benefits, we'll have to see because the Series X is obviously different from a Windows PC.

 

[pjbliverpool]

I 100% think Microsoft have plans to improve Windows I/O - as much as the PC and Windows architecture allows - but this is a mammoth task and there is literally zero evidence of it happening soon, which you would already expect to see signs of.

Apple spent a number of years pathing the way to switch everybody using iOS and macOS to their new APFS filesytsem. This transition began back in 2014 with the introduction of CoreStorage API which allows rock solid logical addressing of storage devices - this is critical for filesystem conversion. After three years of of tweaking CoreStorage, Apple flipped the conversation switch for iOS in 2017 and for macOS for on all-SSD partitions in 2018. Fusion-drives (hybrid SSD/HDD partitions) followed later. Effectively when you installed the latest OS, during an extended install/boot-up, your entire drive was converted to the new filesystem. It was very slick and also terrifying because.. data loss.

Compared to Windows, Apple's hardware ecosystem is very simple and Apple knew exactly hardware and drive controllers were in supported devices. Microsoft are not afforded this luxury and you can be sure that they want a seamless transition without people needing to backup, format and restore.

As for the actual benefits, we'll have to see because the Series X is obviously different from a Windows PC.

What do you think to @chris1515 idea of segregating the read only game data to it's own new file system without touching everything else?

 

[Deleted member 11852]

What do you think to @chris1515 idea of segregating the read only game data to it's own new file system without touching everything else?

You can install non-NTFS filesystems in Windows today, I think EXT3 and EXT4 are fairly well supoorsed as is Apple's APFS! But a solid-state optimised filesystem running on it's own partition, or part of a logical portion on an existing drive would be much quicker to deploy. It doesn't negate the hardware chain but you could cut a lot of the cruft in Window's existing software stack. This approach is actually quite appealing as a Windows gamer because my SSD never feels like it's reaching its potential.

You might have issues with Steam and potentially other/older games that don't play nice with non-NTFS filesystems - this is always the balance, if it's non-standard it may need explicit support. You're probably ok with games going forward but older software may not get fixed.

 

[milk]

How hard is it to create a game centric file system and API to be used exclusively as a secondary PARTITION of your drive, while windows and other apps all sit in good old NTFS?

 

[BRiT]

Relatively simple and also easily able to present the two as one single unified view using something like MergeFS or AuxFS or UnionFS, but those have their own set of quirks on the Linux side. But conceptually it can be done.

Hell, they could even treat this GameFile.Sys like the PageFile.Sys, just allocate huge amount of continuous space on your existing drive after an optimization defrag is run and then treat everything inside that with the new special APIs. Sort of like using VHDs for VMs.

 

[JoeJ]

Game centric and read only could be a missed opportunity.
I think having a fast temporal cache but no guarantee the data is still there when the app launches the next time, would be attractive for many applications, e.g. creative content creation.
MS could lure some loyal Apple apple users, if their Adobe tools suddenly run much faster and cheaper on Windows.

I would be fine with custom locked filesystem, only accessible via certain APIs (plural! hopefully ), but writing would be as important as reading.