General external expansion discussion? *spawn*

Discussion in 'Console Industry' started by goonergaz, Sep 9, 2020.

  1. AzBat

    AzBat Agent of the Bat
    Legend Veteran

    Joined:
    Apr 1, 2002
    Messages:
    7,493
    Likes Received:
    4,360
    Location:
    Alma, AR
    For someone who didn't need it right now(or anytime soon), you're spending a lot of words on it. ;)

    Tommy McClain
     
    Johnny Awesome likes this.
  2. DSoup

    DSoup meh
    Legend Veteran Subscriber

    Joined:
    Nov 23, 2007
    Messages:
    14,907
    Likes Received:
    11,018
    Location:
    London, UK
    I'm genuinely lost. :???: I was trying to help you understand that not everybody thinks like you. It's many counties like America are split right now, regardless of which way you vote, or political leanings, there will always be people who look at things differently.
     
  3. AzBat

    AzBat Agent of the Bat
    Legend Veteran

    Joined:
    Apr 1, 2002
    Messages:
    7,493
    Likes Received:
    4,360
    Location:
    Alma, AR
    Yep, different strokes for different folks. I get it. Keep doing you. Let us come back to this if and when Sony supports external USB hard drives for PS5 games or you run out of internal space. No need to spend cycles on something you don't really care about.

    Tommy McClain
     
    DSoup likes this.
  4. mrcorbo

    mrcorbo Foo Fighter
    Veteran

    Joined:
    Dec 8, 2004
    Messages:
    3,990
    Likes Received:
    2,777
    I get that filesystem differences have to be reconciled, I just don't see why Sony would choose to design their proprietary SSD filesystem in such a way that the work needed to enable moving game data out of it onto external storage would be challenging enough to take significant development time. Just seems like a really weird idea to me.

    Edit: And I failed my pedanticism saving throw, so I now have to point out you can use NTFS on Linux and there's work underway on an OpenZFS port to Windows.
     
    #284 mrcorbo, Nov 8, 2020
    Last edited: Nov 8, 2020
    tuna and AzBat like this.
  5. DSoup

    DSoup meh
    Legend Veteran Subscriber

    Joined:
    Nov 23, 2007
    Messages:
    14,907
    Likes Received:
    11,018
    Location:
    London, UK
    Mark Cerny spent some time explaining why Sony changed things: to remove the bottlenecks. They went to the effort of building a custom I/O processor to boost loading speeds so it would actually be kind of weird if they didn't use a new filesystem (FS) that didn't remove the performance overheads of the old one. :yes:

    Most operating systems allow kernel extensions that support additional non-native FS but the operating system itself doesn't support it, i.e. you generally can't boot from it* (you can do anything with enough work in linux) and none of the underlying OS can manage that FS because the tools are written for the native FS and you'll almost certainly have issues reliably installing/launching apps. Linux and macOS have been able to read/write NTFS for decades but the OS tools are not equipped to maintain the integrity of that FS and can't spot when things become corrupted, nor can they fix the issues.

    Most operating systems make heavy use of extended file attributes and these are generally not standard, even when they are the reason filesystems are different is because of different implementations of data and resource forks.
     
  6. mrcorbo

    mrcorbo Foo Fighter
    Veteran

    Joined:
    Dec 8, 2004
    Messages:
    3,990
    Likes Received:
    2,777
    The issue isn't that it doesn't make sense to create a new filesystem, the issue is that it doesn't make sense to make a new filesystem that didn't account for the need to move game data from it to external storage that may be using a more traditional filesystem and those implementation choices resulting in extra difficulty in realizing that function. This should not be hard unless the people designing the new FS made choices that makes it so. I'd be more inclined to believe Sony just don't consider it a priority to get this feature working than there being any particular difficulty for them to do it.
     
  7. BRiT

    BRiT (>• •)>⌐■-■ (⌐■-■)
    Moderator Legend Alpha

    Joined:
    Feb 7, 2002
    Messages:
    18,795
    Likes Received:
    21,097
    Worst case when moving a game from new filesystem to external is you create an archive file with 2+ files inside of it, one being a signed manifest of files and extended attributes and checksums and then the rest is the data streams. When moving back to new filesystem you validate manifest signing and checksums against data streams, extract the data streams using the extended attributes from the manifest file. If checksums or signing don't match, you don't restore.

    Fun little project even a single junior developer can get started on or could be a single weekend evening project for a senior dev.
     
    arhra likes this.
  8. DSoup

    DSoup meh
    Legend Veteran Subscriber

    Joined:
    Nov 23, 2007
    Messages:
    14,907
    Likes Received:
    11,018
    Location:
    London, UK
    By that logic we shouldn't have filesystems that allow filenames longer than 8-characters with 3-character filetype extensions because you can't copy those files onto FAT16 filesystems. :runaway:

    This is not just hard, this is borderline impossible. Fundamentally if you make changes to your filesystem to best leverage solid state storage, like what Apple did moving from HFS+ to APFS, you accept that there will be features that you can only do on the new filesystem and that interoperability of those functions on older filesystems is limited or missing entirely. Your assertion here this isn't difficult is ludicrous. :yep2:

    I imagine the people designing PS5's filesystem were instructed to make the best filesystem they could with no thought for the legacy extended filesystem because they would support moving PS5 games to it in time.
     
  9. mpg1

    Veteran Newcomer

    Joined:
    Mar 5, 2015
    Messages:
    2,250
    Likes Received:
    1,996
    We are just taking about transferring data from one drive to another, not playing/loading of the game off a different drive. The exact thing the Xbox Series consoles are doing when it comes to external storage for next gen games...

    I could see complication when it comes to the custom internal nvme vs 3rd party off the shelf nvme's for the internal expansion in which a game will be actually be loading/playing off of it. But that's not what's being talked about...
     
    AzBat likes this.
  10. mrcorbo

    mrcorbo Foo Fighter
    Veteran

    Joined:
    Dec 8, 2004
    Messages:
    3,990
    Likes Received:
    2,777
    Of course you can. You just have to rename them first.

    No sale. If implementing APFS on a device were in any way going to limit that device's ability to interact with flash devices or legacy drives they wouldn't implement it.

    Or, it's just a lower priority in the production queue given that, as you said, it's unlikely to present a problem for most users at launch.
     
    #290 mrcorbo, Nov 8, 2020
    Last edited: Nov 8, 2020
  11. DSoup

    DSoup meh
    Legend Veteran Subscriber

    Joined:
    Nov 23, 2007
    Messages:
    14,907
    Likes Received:
    11,018
    Location:
    London, UK
    I like a good "just" thrown into comparing file system! :yes: Let's safety predict that PS5's filesystem has cloned-data forking and manages changes using delta extents, because modern filesystems do. How do you copy those files to a filesystem that does not support those such as NTFS or the extended storage filesystem used by PS4 - and retain all the extensive file attributes? What's your strategy here?

    Just curious from somebody who has worked on filesystems for far too long.
     
  12. mpg1

    Veteran Newcomer

    Joined:
    Mar 5, 2015
    Messages:
    2,250
    Likes Received:
    1,996
    How long do you think it will take for Sony to figure it out?
     
  13. DSoup

    DSoup meh
    Legend Veteran Subscriber

    Joined:
    Nov 23, 2007
    Messages:
    14,907
    Likes Received:
    11,018
    Location:
    London, UK
    So there is a process that needs to be done to prevent dataloss. Now what about if you have more files than you have permutations of filenames using 8.3 names? Or more likely, if you have more files in one directory than the older filesystem can support? What's the plan?

    It's a good job you weren't on Apple's APFS team then, it offer a ton of functionality built into the filesystem that filesystems you may also plug into macOS or iOS do not. Clone forking and delta extents and copy-on-write, which are the basis for more recent versions of TimeMachine. This is how Time Machine can backup a 100Gb drive every hour of every day of every week of every month and not spend 24/7 backing-up if there are tons of changes.

    As far as I know, Sony have not promised storying PS5 games on extended storage drives. Which you are probably aware are not the same format as backup drives for PS4, which is ExFAT. The expectation/hope is that they will support this.
     
    #293 DSoup, Nov 8, 2020
    Last edited: Nov 8, 2020
  14. DSoup

    DSoup meh
    Legend Veteran Subscriber

    Joined:
    Nov 23, 2007
    Messages:
    14,907
    Likes Received:
    11,018
    Location:
    London, UK
    Frankly if I were Sony, I wouldn't even bother. Apple didn't. They have other means to expand storage which is all they ever promised.
     
  15. mrcorbo

    mrcorbo Foo Fighter
    Veteran

    Joined:
    Dec 8, 2004
    Messages:
    3,990
    Likes Received:
    2,777
    Situations you are unlikely to encounter in the types of scenarios we are discussing. Obviously, some attempted operations could fail due to limitations of a legacy filesystem. We are talking about establishing a baseline of functionality here not attempting to enable full interoperability. And given the limited interactions exposed in something like console, your baseline here is even lower than it would be

    And none of this extended functionality prevents you from copying a file from one supported filesystem to another on MacOS/iOS.

    If there really is some crazy metadata/structure difference between Sony's SSD and non-SSD filesystems then just re-containerize the PS5 SSD data for each game when it gets archived. If an entire hard drive can be virtualized into a file, one file or group of files on one should be pretty straightforward when the developer has total control over both ends of the operation.
     
    tinokun likes this.
  16. mrcorbo

    mrcorbo Foo Fighter
    Veteran

    Joined:
    Dec 8, 2004
    Messages:
    3,990
    Likes Received:
    2,777
    It's a good job you are not heading up Sony's software development then!
     
  17. mrcorbo

    mrcorbo Foo Fighter
    Veteran

    Joined:
    Dec 8, 2004
    Messages:
    3,990
    Likes Received:
    2,777
    With a little extra think, I suspect the issue causing the delay/prevention of this functionality is one of system security. Sony may be concerned that this may open up an attack vector against their system and/or content security.
     
    mpg1 likes this.
  18. DSoup

    DSoup meh
    Legend Veteran Subscriber

    Joined:
    Nov 23, 2007
    Messages:
    14,907
    Likes Received:
    11,018
    Location:
    London, UK
    I don't follow this? If you want to inject code into PS5 you can do that now by hacking the binaries of PS4 games on the extended storage, which is where you want to put PS5 games.

    The baseline of functionality is that the external extended storage filesystem has to support the data structures, extended file attributes and resources of the internal one. Otherwise when you copy them, they will just disappear.

    Actually it does. On macOS is you grab a program from the internet that doesn't have a valid developer signature it won't run. You can of course authorise that file to run (it requires admin privs) which sets an extended file attribute which doesn't exist in NTFS or FAT32 or ExFAT. If you copy that program into one of those filesystems then copy it back it'll have lost the authorised flag and will be blocked again and require admin to re-enable it.

    This is exactly what you would need to do but depending on the complexity of the filesystem, writing the export and import tools to capture all the data disassociated data structures and metadata and archive it is not trivial. Especially, as I mentioned above, ideally you'd at least want those stored games to be kept up to date with the latest patches meaning your virtualised game image also needs to be able to grow. Now you're basically writing an entire virtualised filesystem for a f***king games console!
     
    #298 DSoup, Nov 8, 2020
    Last edited: Nov 8, 2020
  19. AzBat

    AzBat Agent of the Bat
    Legend Veteran

    Joined:
    Apr 1, 2002
    Messages:
    7,493
    Likes Received:
    4,360
    Location:
    Alma, AR
    Guys, I would stop posting in this thread. Clearly, Dsoup doesn't care about external USB hard drives on PS5 & you're just wasting your time. ;)

    Tommy McClain
     
  20. DSoup

    DSoup meh
    Legend Veteran Subscriber

    Joined:
    Nov 23, 2007
    Messages:
    14,907
    Likes Received:
    11,018
    Location:
    London, UK
    You couldn't be more wrong.
     
Loading...

Share This Page

  • About Us

    Beyond3D has been around for over a decade and prides itself on being the best place on the web for in-depth, technically-driven discussion and analysis of 3D graphics hardware. If you love pixels and transistors, you've come to the right place!

    Beyond3D is proudly published by GPU Tools Ltd.
Loading...