General external expansion discussion? *spawn*

[AzBat]

For someone who didn't need it right now(or anytime soon), you're spending a lot of words on it.

Tommy McClain

 

[Deleted member 11852]

For someone who didn't need it right now(or anytime soon), you're spending a lot of words on it.

I'm genuinely lost. I was trying to help you understand that not everybody thinks like you. It's many counties like America are split right now, regardless of which way you vote, or political leanings, there will always be people who look at things differently.

 

[AzBat]

Yep, different strokes for different folks. I get it. Keep doing you. Let us come back to this if and when Sony supports external USB hard drives for PS5 games or you run out of internal space. No need to spend cycles on something you don't really care about.

Tommy McClain

 

[mrcorbo]

You can copy files between filesystems when they share the the same or equivalent metadata, permissions and resource fork structures. When they don't, you can loose critical metadata that the filesystem may not report and you won't know you've lost the data until you copy files back and metadata has disappeared. If you're not familiar with the differences between filesystems it's not something I can easily explain here, it's a complex issue. It's the same reason you can't use NTFS for linux or macOS and you can't use ZFS for Windows. Filesystems are different.

I get that filesystem differences have to be reconciled, I just don't see why Sony would choose to design their proprietary SSD filesystem in such a way that the work needed to enable moving game data out of it onto external storage would be challenging enough to take significant development time. Just seems like a really weird idea to me.

Edit: And I failed my pedanticism saving throw, so I now have to point out you can use NTFS on Linux and there's work underway on an OpenZFS port to Windows.

 

[Deleted member 11852]

I get that filesystem differences have to be reconciled, I just don't see why Sony would choose to design their proprietary SSD filesystem in such a way that the work needed to enable moving game data out of it onto external storage would be challenging enough to take significant development time. Just seems like a really weird idea to me.

Mark Cerny spent some time explaining why Sony changed things: to remove the bottlenecks. They went to the effort of building a custom I/O processor to boost loading speeds so it would actually be kind of weird if they didn't use a new filesystem (FS) that didn't remove the performance overheads of the old one.

Edit: And I failed my pedanticism saving throw, so I now have to point out you can use NTFS on Linux and there's work underway on an OpenZFS port to Windows.

Most operating systems allow kernel extensions that support additional non-native FS but the operating system itself doesn't support it, i.e. you generally can't boot from it* (you can do anything with enough work in linux) and none of the underlying OS can manage that FS because the tools are written for the native FS and you'll almost certainly have issues reliably installing/launching apps. Linux and macOS have been able to read/write NTFS for decades but the OS tools are not equipped to maintain the integrity of that FS and can't spot when things become corrupted, nor can they fix the issues.

Most operating systems make heavy use of extended file attributes and these are generally not standard, even when they are the reason filesystems are different is because of different implementations of data and resource forks.

 

[mrcorbo]

Mark Cerny spent some time explaining why Sony changed things: to remove the bottlenecks. They went to the effort of building a custom I/O processor to boost loading speeds so it would actually be kind of weird if they didn't use a new filesystem (FS) that didn't remove the performance overheads of the old one.

The issue isn't that it doesn't make sense to create a new filesystem, the issue is that it doesn't make sense to make a new filesystem that didn't account for the need to move game data from it to external storage that may be using a more traditional filesystem and those implementation choices resulting in extra difficulty in realizing that function. This should not be hard unless the people designing the new FS made choices that makes it so. I'd be more inclined to believe Sony just don't consider it a priority to get this feature working than there being any particular difficulty for them to do it.

 

[BRiT]

Worst case when moving a game from new filesystem to external is you create an archive file with 2+ files inside of it, one being a signed manifest of files and extended attributes and checksums and then the rest is the data streams. When moving back to new filesystem you validate manifest signing and checksums against data streams, extract the data streams using the extended attributes from the manifest file. If checksums or signing don't match, you don't restore.

Fun little project even a single junior developer can get started on or could be a single weekend evening project for a senior dev.

 

[Deleted member 11852]

The issue isn't that it doesn't make sense to create a new filesystem, the issue is that it doesn't make sense to make a new filesystem that didn't account for the need to move game data from it to external storage that may be using a more traditional filesystem and those implementation choices resulting in extra difficulty in realizing that function.

By that logic we shouldn't have filesystems that allow filenames longer than 8-characters with 3-character filetype extensions because you can't copy those files onto FAT16 filesystems.

This should not be hard unless the people designing the new FS made choices that makes it so. I'd be more inclined to believe Sony just don't consider it a priority to get this feature working than there being any particular difficulty for them to do it.

This is not just hard, this is borderline impossible. Fundamentally if you make changes to your filesystem to best leverage solid state storage, like what Apple did moving from HFS+ to APFS, you accept that there will be features that you can only do on the new filesystem and that interoperability of those functions on older filesystems is limited or missing entirely. Your assertion here this isn't difficult is ludicrous.

I imagine the people designing PS5's filesystem were instructed to make the best filesystem they could with no thought for the legacy extended filesystem because they would support moving PS5 games to it in time.

 

[mpg1]

We are just taking about transferring data from one drive to another, not playing/loading of the game off a different drive. The exact thing the Xbox Series consoles are doing when it comes to external storage for next gen games...

I could see complication when it comes to the custom internal nvme vs 3rd party off the shelf nvme's for the internal expansion in which a game will be actually be loading/playing off of it. But that's not what's being talked about...

 

[mrcorbo]

By that logic we shouldn't have filesystems that allow filenames longer than 8-characters with 3-character filetype extensions because you can't copy those files onto FAT16 filesystems.

Of course you can. You just have to rename them first.

This is not just hard, this is borderline impossible. Fundamentally if you make changes to your filesystem to best leverage solid state storage, like what Apple did moving from HFS+ to APFS, you accept that there will be features that you can only do on the new filesystem and that interoperability of those functions on older filesystems is limited or missing entirely. Your assertion here this isn't difficult is ludicrous.

No sale. If implementing APFS on a device were in any way going to limit that device's ability to interact with flash devices or legacy drives they wouldn't implement it.

I imagine the people designing PS5's filesystem were instructed to make the best filesystem they could with no thought for the legacy extended filesystem because they would support moving PS5 games to it in time.

Or, it's just a lower priority in the production queue given that, as you said, it's unlikely to present a problem for most users at launch.

 

[Deleted member 11852]

We are just taking about transferring data from one drive to another, not playing/loading of the game off a different drive. The exact thing the Xbox Series consoles are doing when it comes to external storage for next gen games...

I like a good "just" thrown into comparing file system! Let's safety predict that PS5's filesystem has cloned-data forking and manages changes using delta extents, because modern filesystems do. How do you copy those files to a filesystem that does not support those such as NTFS or the extended storage filesystem used by PS4 - and retain all the extensive file attributes? What's your strategy here?

Just curious from somebody who has worked on filesystems for far too long.

 

[mpg1]

I like a good "just" thrown into comparing file system! Let's safety predict that PS5's filesystem has cloned-data forking and manages changes using delta extents, because modern filesystems do. How do you copy those files to a filesystem that does not support those such as NTFS or the extended storage filesystem used by PS4 - and retain all the extensive file attributes? What's your strategy here?

Just curious from somebody who has worked on filesystems for far too long.

How long do you think it will take for Sony to figure it out?

 

[Deleted member 11852]

Of course you can. You just have to rename them first.

So there is a process that needs to be done to prevent dataloss. Now what about if you have more files than you have permutations of filenames using 8.3 names? Or more likely, if you have more files in one directory than the older filesystem can support? What's the plan?

No sale. If implementing APFS on a device were in any way going to limit that device's ability to interact with flash devices or legacy drives they wouldn't implement it.

It's a good job you weren't on Apple's APFS team then, it offer a ton of functionality built into the filesystem that filesystems you may also plug into macOS or iOS do not. Clone forking and delta extents and copy-on-write, which are the basis for more recent versions of TimeMachine. This is how Time Machine can backup a 100Gb drive every hour of every day of every week of every month and not spend 24/7 backing-up if there are tons of changes.

Or, it's just a lower priority in the production queue given that, as you said, it's unlikely to present a problem for most users at launch.

As far as I know, Sony have not promised storying PS5 games on extended storage drives. Which you are probably aware are not the same format as backup drives for PS4, which is ExFAT. The expectation/hope is that they will support this.

 

[Deleted member 11852]

How long do you think it will take for Sony to figure it out?

Frankly if I were Sony, I wouldn't even bother. Apple didn't. They have other means to expand storage which is all they ever promised.

 

[mrcorbo]

So there is a process that needs to be done to prevent dataloss. Now what about if you have more files than you have permutations of filenames using 8.3 names? Or more likely, if you have more files in one directory than the older filesystem can support? What's the plan?

Situations you are unlikely to encounter in the types of scenarios we are discussing. Obviously, some attempted operations could fail due to limitations of a legacy filesystem. We are talking about establishing a baseline of functionality here not attempting to enable full interoperability. And given the limited interactions exposed in something like console, your baseline here is even lower than it would be

It's a good job you weren't on Apple's APFS team then, it offer a ton of functionality built into the filesystem that filesystems you may also plug into macOS or iOS do not. Clone forking and delta extents and copy-on-write, which are the basis for more recent versions of TimeMachine. This is how Time Machine can backup a 100Gb drive every hour of every day of every week of every month and spend 24/7 backup up even if there are tons of changes.

And none of this extended functionality prevents you from copying a file from one supported filesystem to another on MacOS/iOS.

As far as I know, Sony have not promised storying PS5 games on extended storage drives. Which you are probably aware are not the same format as backup drives for PS4, which is ExFAT. The expectation/hope is that they will support this.

If there really is some crazy metadata/structure difference between Sony's SSD and non-SSD filesystems then just re-containerize the PS5 SSD data for each game when it gets archived. If an entire hard drive can be virtualized into a file, one file or group of files on one should be pretty straightforward when the developer has total control over both ends of the operation.

 

[mrcorbo]

Frankly if I were Sony, I wouldn't even bother. Apple didn't. They have other means to expand storage which is all they ever promised.

It's a good job you are not heading up Sony's software development then!

 

[mrcorbo]

With a little extra think, I suspect the issue causing the delay/prevention of this functionality is one of system security. Sony may be concerned that this may open up an attack vector against their system and/or content security.

 

[Deleted member 11852]

With a little extra think, I suspect the issue causing the delay/prevention of this functionality is one of system security. Sony may be concerned that this may open up an attack vector against their system and/or content security.

I don't follow this? If you want to inject code into PS5 you can do that now by hacking the binaries of PS4 games on the extended storage, which is where you want to put PS5 games.

We are talking about establishing a baseline of functionality here not attempting to enable full interoperability. And given the limited interactions exposed in something like console, your baseline here is even lower than it would be

The baseline of functionality is that the external extended storage filesystem has to support the data structures, extended file attributes and resources of the internal one. Otherwise when you copy them, they will just disappear.

And none of this extended functionality prevents you from copying a file from one supported filesystem to another on MacOS/iOS.

Actually it does. On macOS is you grab a program from the internet that doesn't have a valid developer signature it won't run. You can of course authorise that file to run (it requires admin privs) which sets an extended file attribute which doesn't exist in NTFS or FAT32 or ExFAT. If you copy that program into one of those filesystems then copy it back it'll have lost the authorised flag and will be blocked again and require admin to re-enable it.

If there really is some crazy metadata/structure difference between Sony's SSD and non-SSD filesystems then just re-containerize the PS5 SSD data for each game when it gets archived.

This is exactly what you would need to do but depending on the complexity of the filesystem, writing the export and import tools to capture all the data disassociated data structures and metadata and archive it is not trivial. Especially, as I mentioned above, ideally you'd at least want those stored games to be kept up to date with the latest patches meaning your virtualised game image also needs to be able to grow. Now you're basically writing an entire virtualised filesystem for a f***king games console!

 

[AzBat]

Guys, I would stop posting in this thread. Clearly, Dsoup doesn't care about external USB hard drives on PS5 & you're just wasting your time.

Tommy McClain

 

[Deleted member 11852]

Guys, I would stop posting in this thread. Clearly, Dsoup doesn't care about external USB hard drives on PS5 & you're just wasting your time.

You couldn't be more wrong.