Alternative distribution to optical disks : SSD, cards, and download*

[taisui]

Preliminary install aside - this was discussed earlier. Search for the discussion on "40 seconds". Discussion then moved on to what happens after that.

I replied with a link that the X1 just can't do partial install now, and MSFT is aware of the issue.

Then you replied and start claiming that somehow the quick boot up time has to do with the zlib hardware, and kept going back what Cerny said, which you had completely misunderstood.
And now you are saying that it wasn't what you were saying?

 

[Brad Grenz]

Malarkey...

When PS4 "installs" a game, it does it in the OS, where it copies the chunks required to boot up into the HDD first. Once it's ready, the icon switch to "Start" when you can start the game [from the HDD] and in the background it keeps copying.

If it was able to load straight it to the RAM AND copy to the HDD concurrently, then why would you need to even wait for it to install in the OS first? It ought to just allow you to "Start" immediately upon inserting the BD, with the zlib voodoo doing all the work.

Because there would be nothing to look at in game for about that long? Leaving you in the main UI while data is being cached to RAM makes more sense than launching a blank screen you have to stare at for 40 seconds.

 

[Deleted member 11852]

I replied with a link that the X1 just can't do partial install now, and MSFT is aware of the issue.

Then you replied and start claiming that somehow the quick boot up time has to do with the zlib hardware.
And now you are saying that it wasn't what you were saying?

I've not been talking about the Xbox One at all. I don't own one, don't know anybody with one and have never used one. I am only talking about my experience with the PlayStation 4, which I do own, and throwing in what Mark Cerny has said about PlayGo. Everything I've said is about PlayGo. Which is why I keep mentioning PlayGo.

But the install process and hardware zlib aren't mutually exclusive. If you bother to read the interview with Mark Cerny, he explains it here.

However, PlayGo "is two separate linked systems," Cerny said. The other is to do with the Blu-ray drive -- to help with the fact that it is, essentially, a bit slow for next-gen games.

"So, what we do as the game accesses the Blu-ray disc, is we take any data that was accessed and we put it on the hard drive. And if then if there is idle time, we go ahead and copy the remaining data to the hard drive. And what that means is after an hour or two, the game is on the hard drive, and you have access, you have dramatically quicker loading... And you have the ability to do some truly high-speed streaming."

To further help the Blu-ray along, the system also has a unit to support zlib decompression -- so developers can confidently compress all of their game data and know the system will decode it on the fly. "As a minimum, our vision is that our games are zlib compressed on media," said Cerny.

 

[taisui]

I've not been talking about the Xbox One at all.

Ya ya ya, whatever, when I said it's just software, you relied:

It's partly hardware; the PS4's hardware zlib decompression plays a role. I believe Xbox One has something similar?

and kept quoting the same sentences again and again while not understanding what Cerny is saying, is not going to help your case, keep on trucking.

 

[Deleted member 11852]

Ya ya ya, whatever, when I said it's just software, you relied:

Let's recap. The discussion noted the slow install times on Xbox One then moved on to how quick it was on PlayStation 4. You posted that Microsoft was aware of the issue and that it's just software.

I replied saying that it's also a hardware issue. I was referring to PlayStation 4's implementation here and quoted Mark Cerny talking about PlayGo and the use of zlib decompression hardware, which I believe is partially to credit for fast installs on PlayStation 4. Anticipating this may be part of Microsoft's solution I asked if Xbox One also had this - I thought it had but wasn't certain.

and kept quoting the same sentences again and again while not understanding what Cerny is saying, is not going to help your case, keep on trucking.

I understand what Cerny is saying, it's very clear. And for the avoidance of doubt, this is my understanding of PlayGo working with a Blu-ray disc.

1. User inserts disc and the preliminary mandatory install begins. 2. When the minimum install has occurred, game can be launched and will run from HDD unless the game needs data from the Blu-ray, at which point that data is loaded from disc and also copied to the HDD. 3. When the game is not accessing the disc, the remainder of the game is copied (installed) to the HDD.

If you're only been talking about Xbox One then we're obviously talking about different things.

 

[Shifty Geezer]

Ya ya ya, whatever, when I said it's just software, you relied:

and kept quoting the same sentences again and again while not understanding what Cerny is saying, is not going to help your case, keep on trucking.

Can we ditch the bickering and turn this into an adult conversation? You may want to clarify your argument. I'm thinking that you're saying PS4 has not got a hardware advantage over XB1 for partial installs where you see DSoup as advocating this position. If so, the argument can be made a lot stronger by talking about the hardwares rather than talking about one's choice of sentences.

It's certainly possible that PS4 is architected to support 'through writes' to HDD from BRD. That little support chip that installs from downloads could, and should for consistency, be working the same from a disc as from a download, the difference being the disc represents a ~200mbps download speed. It might have enough cache for file IO without main RAM needing to be involved. It's a bit of an unknown at this point, unless I've missed some details in the tear-downs.

As such, the PS4 PlayGo install system would be designed to download the starting amount of data. This download will be too slow for realtime loading in the beginning. 1 GB may take, I dunno, 10 minutes. You can't have the game loading for 10 minutes and very slowly building up the main interface - it'd look ridiculous. So instead, you force a wait on the system menu to preload the first piece of the game.

Using exactly that same system, you'd precache the initial beginnings of game from BRD, only it'd take 40 seconds instead of 10 minutes. Then it's ready to launch. There's no reason to do it any other way. This way requires only one solution and is homogenous. Direct play from disc like a PS2 would require a second load and install system which just isn't necessary.

So, a 40 second initial install makes sense. The question then becomes whether the data is loaded from disk into RAM and then copied out to HDD, or copied to HDD and RAM at the same time, or copied to HDD independently from RAM and loaded from there via the game. I don't see how those particulars matter at all. It works and they're all much of a muchness.

XB1's requirement for larger installs seems to be a software issue as some titles have partial installs. That suggests either a manual implementation is needed by devs, or there are caveats.

 

[-tkf-]

I thought that typical game assets were either compressed beforehand, like textures etc or not very compress friendly. Using a hardware solution also gives us a nice little read boost from the harddrive since filling up 2GB might just involve reading 1GB with a 2:1 ratio..

 

[Npl]

Textures and sound aside, model and level data could very well appear to be loaded "uncompressed" and maybe even "ready to use" behind the zlib hardware.
Think of files that are paged from harddisk directly to ram. You could then walk the structure of a big city/universe/multimegaultraverse directly from ram, the OS will transparently load the missing data and purge out the ones you havent touched in a while.
64bit addresspace should be ample for those kinda things, fixed hardware would allow you to "bake in" pointers, the compression is handled by hardware specialised for it and Im sure lazy gamedevs would love it

 

[Blazkowicz]

Most of those things should be handled by the kernel and the MMU I think and disk caching is old hat.

Yes indeed with the address space you can have any file memory mapped. The wikipedia article on that seems decent, talks of "magical" things that are speculated here but that I guess have existed already for 20 years or more.

Talk about compression is done with every console gen since the N64 days. Slightly before it, Donkey Kong Country on the SNES decompressed graphical assets on the fly thanks to a special chip (and the base console supports ADPCM digital sound). Maybe that's similar to using S3TC textures.

 

[Npl]

Yeah, dint say anything is new at all, its just nice and comfortable to use if you have 64bit, fixed addresses for file (unlike generic OSes pointer could already be initialized) and a companion cpu + hardware for decompression.
Games still wont like the erratic delays, so its not as this would be the end to custom solutions

 

[taisui]

So, a 40 second initial install makes sense. The question then becomes whether the data is loaded from disk into RAM and then copied out to HDD, or copied to HDD and RAM at the same time, or copied to HDD independently from RAM and loaded from there via the game. I don't see how those particulars matter at all. It works and they're all much of a muchness.

Sorry for asking in a technical forum
AFAIK the games are running off the HDD, which is why some modes are locked away upon the first boot until the full installation is complete.
Between that and how people was able to improve loading performance by going to faster HDD/SSD, it's pretty apparent how the data flow from and to at when.

The fact that there are multiple "theories" of how people describing how this works, from the pre-loading into RAM, to concurrent reading with zlib, to the possible free mirroring...
It means that no one actually knows how it works. Just speculations.

or Magic. Can't blame those who actually believe that you can install 40Gs of data under a minute

 

[Shifty Geezer]

I have no idea what you're talking about. AFAICS no-one's claimed a full 40GB install in 40 seconds. It's always been about the partial installs and PS4 running a game from a small piece of the whole thing from disc (or download) where XB1 is needing more to be installed.

It means that no one actually knows how it works. Just speculations.

The thread topic is alternative distribution and talking about time to copy a game off a medium. PS4 and XB1 were raised to compare solutions, with PS4 proving there exists a fast partial-install system. How it goes about that isn't necessary information to continue the discussion on game media and downloads/installs. It's also not necessary knowledge for any gamer or dev as it works transparently in the background - devs would just need to know how to structure their distribution to use the system.

Ergo, speculations are all we need in this thread. There are enough ideas to show workable solutions for XB1 and future platforms. If people want to get into the nitty-gritty of why PS4's initial installs are lower than XB1s, that discussion deserves its own thread.

 

[Warchild]

I ran my ps4 ssd through acronis to see how the ps4 formats stuff

Here is my normal hdd windows partition

I'm trying to find the cluster size the ps4 uses but it only shows sectors per cluster which is 57. Any way of knowing?

 

[Shifty Geezer]

That's certainly an original file system name. Edit: If you run that name into a Japanese font, do you get a Japanese name? I suppose realistically there's no way of knowing, because you will get some characters and if they're random, we may not tell due to the interesting approach the Japanese often take to naming things. "Happy birthright within prism" may be gibberish or may actually be the name of the file system.

 

[pcchen]

Judging from the weird sector size and cluster size, I'd say that the partition is not in a format that the application recognises. That explains why the file system name and other information (such as the number of total sectors) are gibberish.

 

[Shifty Geezer]

You don't think the PS4's HDD has 1.4 quintillion sectors? Hmm, you may be onto something there...

 

[MrFox]

It could be a block device encryption? That would make everything gibberish, including the partition's sector 0.

The PS4 is using a modified FreeBSD 9.0, so probably the default UFS file system, which have a default fragment size of 4k and a block size of 32k. But if it's encrypted, we'll never know.

 

[Deleted member 11852]

It could be a block device encryption? That would make everything gibberish, including the partition's sector 0.

I'll have a look at the weekend but yeah, it could be using a logical encrypted volume. It'll be easy enough to tell. Not only is such an approach more secure but the encrypt/decrypt implementation is a constant and doesn't impact fragmented file I/O, it's just a mild overhead of the filesystem.

Jaguar supports AES-NI so that would be the way to go.

 

[Warchild]

That's certainly an original file system name. Edit: If you run that name into a Japanese font, do you get a Japanese name? I suppose realistically there's no way of knowing, because you will get some characters and if they're random, we may not tell due to the interesting approach the Japanese often take to naming things. "Happy birthright within prism" may be gibberish or may actually be the name of the file system.

Yeah i think i tried that and random characters still came up.

I'll have a look at the weekend but yeah, it could be using a logical encrypted volume. It'll be easy enough to tell. Not only is such an approach more secure but the encrypt/decrypt implementation is a constant and doesn't impact fragmented file I/O, it's just a mild overhead of the filesystem.

Jaguar supports AES-NI so that would be the way to go.

Be careful, make sure you back up your gamesaves to a usb because the moment you scan the hdd on your computer and insert it back in the ps4 you will have to format it again. It happened to me.

 

[Deleted member 11852]

Be careful, make sure you back up your gamesaves to a usb because the moment you scan the hdd on your computer and insert it back in the ps4 you will have to format it again. It happened to me.

Thanks the warning - all my game saves are in Sony's cloud Hopefully I won't have any problems, I'll be using a hardware disc duplicator - it's a PC system designed to clone secure HDDs (BeCrypt, FoRSA etc) but one of the cool things is that you can [strike]mess with[/strike] analyse the filesytem from the source drive.