One could use... fractal (yep, this sounds hip) image compression to get acceptable download size, then de- and recompress the currently played level to DXT, cache that to SSD, and call it uncompressed, ready to stream data.
Sony's ReRAM plans - what can and can't ReRAM bring to a console? *spawn
[Q
Not talking about totally uncompressed data especially textures. Compressed textures remain as is.
If what Cerny said about raw bandwidth is true, then we might be looking at 4gb/s upward of data streamed. So you think this speed is not an overkill? Can you decompress that amount of data in real-time? If not then what's the point. If Sony is hitting for big bandwidth then they either has: a. A capable decompressor that will not render the bandwidth overkill b. They are streaming data that is ready to be read.
If it is (b) then cache system makes a lot of sense unless you want an SSD with 2-3 games installed.
Also I'd like you to look at what Remedy dev said. Is there at any point, any scenerio, any chance that an SSD with 2gb/s - 4gb/s (decompressed in real-time) will result in long load times?
Not talking about totally uncompressed data especially textures. Compressed textures remain as is.
If what Cerny said about raw bandwidth is true, then we might be looking at 4gb/s upward of data streamed. So you think this speed is not an overkill? Can you decompress that amount of data in real-time? If not then what's the point. If Sony is hitting for big bandwidth then they either has: a. A capable decompressor that will not render the bandwidth overkill b. They are streaming data that is ready to be read.
If it is (b) then cache system makes a lot of sense unless you want an SSD with 2-3 games installed.
Also I'd like you to look at what Remedy dev said. Is there at any point, any scenerio, any chance that an SSD with 2gb/s - 4gb/s (decompressed in real-time) will result in long load times?
Woah.. back up there, sunshine. You can't just waltz in here making valid points and observations.
Damn n00bs.
Please read the conversation first. Can save you from insulting people.
It's a safe bet that data will be aranged and compressed very differently the comming gen.
Basically, different bottlenecks call for different choices. High compression is an answer to limited storage and/or bandwith. This gen devs have shown to care little about storage requirements, so it was mainly a HDD bandwith consideration then. Once that's not a bottleneck anymore, devs will reconsider their choices.
Currently most games pack in data in large blocks containing multiple types of data all heavily compressed. There is usually a lot of redundant data in different blocks to reduce seeking on the HDD which is the biggest latency killer there. High compression rates are meant to combat slow read speeds + latency.
The big win of SSD is low latency seek, so data can be requested very granularly at similar speeds of requesting large continuous blocks. So space will be saved thanks to less redundancy. But for that to work, you can't compress a bunch of stuff together. Data will be arranged in many more smaller blocks to make the most of the SSD (and the relatively small memory pool)
But on the other hand, once devs realise they have high-speed access to data, and very efficiently so, they might say "Why am I spending so much CPU time on decompression and not on next-gen physics when I can just leave data less compressed on the SSD and let the amazing SSD read speed take care of it?"
Basically, different bottlenecks call for different choices. High compression is an answer to limited storage and/or bandwith. This gen devs have shown to care little about storage requirements, so it was mainly a HDD bandwith consideration then. Once that's not a bottleneck anymore, devs will reconsider their choices.
Currently most games pack in data in large blocks containing multiple types of data all heavily compressed. There is usually a lot of redundant data in different blocks to reduce seeking on the HDD which is the biggest latency killer there. High compression rates are meant to combat slow read speeds + latency.
The big win of SSD is low latency seek, so data can be requested very granularly at similar speeds of requesting large continuous blocks. So space will be saved thanks to less redundancy. But for that to work, you can't compress a bunch of stuff together. Data will be arranged in many more smaller blocks to make the most of the SSD (and the relatively small memory pool)
But on the other hand, once devs realise they have high-speed access to data, and very efficiently so, they might say "Why am I spending so much CPU time on decompression and not on next-gen physics when I can just leave data less compressed on the SSD and let the amazing SSD read speed take care of it?"
Hahah, this was yesterday:
x-------------x
loading...
x-------------x
this you get tomorrow:
x----------------------------------------------x
decompressing...........................
x----------------------------------------------x
Storage speed is not the problem. Capacity is.
Either this or keep going with looking at 1000 instances of the same Quixel rock.
(ok, guess it's the latter for next gen)
x-------------x
loading...
x-------------x
this you get tomorrow:
x----------------------------------------------x
decompressing...........................
x----------------------------------------------x
Storage speed is not the problem. Capacity is.
Either this or keep going with looking at 1000 instances of the same Quixel rock.
(ok, guess it's the latter for next gen)
D
Deleted member 11852
Guest
Please read the conversation first. Can save you from insulting people.
My rapidly dwindling braincells are testament that I am reading what some people are writing. What I don't think it happening is people are reading what they write, or at least not thinking about it for more than a second.
Heh. That's today's PC with NVMe. And that will be tomorrow's console if ssd speed outrun decompression speed.
And that's what the cache system will potentially solve, no?
On another note, haven't I laid out my points clearly enough? Maybe I lack the technical jargon to explain it but I thought it's pretty understandable.
Not necessarily.
In a game there is no constant load on CPU systems. Physics simulation, animation, audio, all those systems will have spikes in runtime, especially when there is much action happening (fight with some enemies).
But that's the situation where you want to keep your fps target the most, so you make this worst case scenario your limit (or, at least you should).
And this means you have plenty of time for decompression tasks running in the background, most of the time.
So you can load and decompress the area surrounding the player during gameplay, and there is no urgent need for a dedicated decompression chip or anything fancy.
Next gen has powerful CPU, so this should work without flaws now i guess.
The only thing that can solve this problem for now is progress in compression software. Having fast streaming is a minor problem in comparison.
Hardware can only help to give some linear speed ups. But software can achieve exponential factors. Be it performance, or compression ratios, whatever, if we are lucky. But first figure out what works, what's needed, and after that think about tailored hardware solutions - if necessary.
For the moment, reusing texture and geometry multiple times is state of the art and people are so used to it nobody complains or tries to improve beyond this so badly. Just fools like i do
What we do here in this thread seems the opposite: Bring in some new hardware and then brainstorm how games could benefit.
In the best case this could indeed spur some ideas / inspiration, but it may be totally unrelated to the initial proposed hardware.
(New hardware always brings in additional costs - money, power, chip area. That's why i'm usually against it - i prefer more raw general purpose compute power instead almost always. Just to explain my resistance.)
But you have to have some space to store that less compressed data or are you suggesting that the entire game is in that state when it's installed in the SSD?
I was reading about virtual RAM and didn't realize that all the things I'm basically suggesting is like virtual RAM using SSD, but perhaps with better I/O than what is possible on PC. The problem with virtual RAM in a game console would be the endurance of the SSD. I mean every game will write to it if it's enabled to developers. Can the TLC SSD endure a constant write to it if a portion of it is used as virtual RAM?
To be fair, I am trying to make sense of the "raw bandwidth higher than what's available on PC". Is there a reason for Sony to go for such bandwidth when the problem is not bandwidth but decompression?
And how do you make sense of what the Remedy dev said. Is there any point or any scenario that a long load times will happen in a 2 - 4gb/s SSD?
Because that's the best development practice for what's available, no?
I will not contest that. But how do you make sense of this statement by a Remedy dev:
"When it comes to the PS5, faster hardware is always appreciated and will make life easier in the short term, but it's the new SSD that really stands out; essentially streaming will become something that we don't really have to worry so much about and it will free up some extra CPU bandwidth in the process," said Remedy's lead programmer Sean Donnelly while speaking to the Official PlayStation Magazine.
?
If commonly compressed data will remain compressed then whats the complaint?
Compression isn’t an all or nothing affair. You just target the data that makes sense for compression. Just as it’s used now.
The problem today is that memory systems can’t keep up with computational systems. Data movement is the bottleneck. Compressing data is just a way to lessen the effects of that bottleneck.
If nvidia and AMD are effectively using compression where bandwidth runs into the 100s of GBps, how will it create a bottleneck where GBps of bandwidth drops to single digits?
That doesn’t necessarily mean running without compression.
Last edited:
Oh ok. That's where the difference lies. Can you take a look at this then.
What's the new bottleneck with regards to load times?
Some answers given:
Do you not agree to this?
We have been talking about less compression or better algorithm to take advantage of the ratio and speed, or better yet no compression at all. But it was suggested that storage space will be a problem.
I was replying to the decompression chip comment.
It depends.
Spend time optimizing for frame rate, or spend time optimizing loading times?
How much data has your game?
Is it open world, or sequential levels?
What happens on player death - need to reload everything? Can just respawn in persistent world ?
How often will the player die?
What else do you need to do, except loading data / decompression? Presimulating a whole world somehow?
...
The question still stands. Why do you need large bandwidth which Sony is shooting for if your CPU can't keep up. That was my reason for bringing that reddit discussion up.
I mean even some of the comments there were suggesting to decompress the data on their NVMe SSD, because the problem is not the bandwidth, it's decompression.
Also, I've been hearing that the whole point of this ultra-fast SSD is to have streaming really fast that it will free some RAM caching, specially in open-worlds. If you still have to go through the processing of decompressing the data then you have a problem of being limited with the amount of data you can stream or you sacrifice a lot of your CPU time. That's how its done today, maybe that's still how it'll be done tomorrow. So again the question, what's the point of that high bandwidth when you're limited by decompression.
This could happen, but it does not have to.
I assume they have something like a DMA engine in discrete GPUs, which transfers the data without needing compute resources. So likely they 'page' textures and geometry directly to graphics ram. Neither CPU nor GPU take a hit.
Texture, geometry and audio, also animation, all this can be used with it's compression as is, and that's the most data usually.
So no decompression is needed in the regular case.
The confusion i have maybe added here is about another problem: Storage space in case you want... "Unlimited Detail"
You surely know there are compression methods that decompress quickly, or even in hardware like DXT, but those have low compression ratios.
Then there are methods with much higher ratios like Jpeg, but to use the data you have to un- and recompress (Jpeg->DXT) at runtime which has a cost.
The former is the norm, the latter is rarely used in games, or just for some of the data. (e.g. Rage which used Jpeg for textures, which adds additional costs to the streaming system)
But even in this case you benefit from SSD of course.
The overkill example would be to cache your uncompressed Jpegs as DXT on SSD like my fractal post, but it was not meant so seriously. It would make sense if decompression becomes extremely costly, but mostly it's just the compression that takes long.
BTW, there is efford for something better than Jpeg-DXT, for example https://opensource.googleblog.com/2019/05/google-and-binomial-partner-to-open.html