Current Generation Hardware Speculation with a Technical Spin [post GDC 2020] [XBSX, PS5]

Discussion in 'Console Technology' started by Proelite, Mar 16, 2020.

Thread Status:
Not open for further replies.
  1. dobwal

    Legend Veteran

    Joined:
    Oct 26, 2005
    Messages:
    5,647
    Likes Received:
    1,836
    You not losslessly compressing individual blocks. You are losslessly compressing individual tiles of blocks. Each tile is composed of multiple blocks of lossy compressed pixels.

    You can selectively load each tile into VRAM but you won't have the ability to randomly select pixel blocks within each tile so you need to decompress first and just keep the pixels in the texture block compression format while in VRAM.
     
    #2721 dobwal, May 29, 2020
    Last edited: May 29, 2020
  2. iroboto

    iroboto Daft Funk
    Legend Regular Subscriber

    Joined:
    Mar 6, 2014
    Messages:
    12,793
    Likes Received:
    15,308
    Location:
    The North
    Will need to see if either format is capable of this. You would need a different compression algorithm imo. I don't think Kraken is designed for incremental streaming AFAIK or random access. BCPack is unknown since it's designed for textures, but I'm still unsure as to what that means.
     
    PSman1700 likes this.
  3. dobwal

    Legend Veteran

    Joined:
    Oct 26, 2005
    Messages:
    5,647
    Likes Received:
    1,836
    I think you are misunderstanding.

    Kraken or Zlib is performing lossless compression. It’s simply serving as a way to minimize bandwidth consumption between the ssd and vram as well as the capacity used within the SSD. Textures have an additional layer of compression thats lossy and is a block based.

    The block based compression also helps with bandwidth consumption across the ssd to vram and minimize storage usage on the ssd. But it also serves to minimize VRAM usage as data is compressed while allowing random access.

    This layered or two-stage compression approach has been available for quite some time. It’s not really new. In fact there are newer derivative approaches that allows better compression.

    http://www.jacobstrom.com/publications/StromWennerstenHPG2011.pdf
     
    #2723 dobwal, May 29, 2020
    Last edited: May 29, 2020
  4. DSoup

    DSoup meh
    Legend Veteran Subscriber

    Joined:
    Nov 23, 2007
    Messages:
    14,727
    Likes Received:
    10,874
    Location:
    London, UK
    This is the dilemma with the PC, many of the individual components and bus controllers have their own driver and are connected to other components over a bus. Consoles are different in that there is effective one I/O bus to unified RAM pool from which the CPU and GPU bus can access the pool.

    Kraken is optimised for images, Zlib is not. PS5 has hardware decompressors for both. Likewise XSX supports BCPack and Zlib.
     
    Lucid_Dreamer and function like this.
  5. Shifty Geezer

    Shifty Geezer uber-Troll!
    Moderator Legend

    Joined:
    Dec 7, 2004
    Messages:
    43,576
    Likes Received:
    16,033
    Location:
    Under my bridge
    That's understood,. The concern is if the Kraken-type compression on SSD allow PRT to be read. If not compressed beyond DXTC, we could store and read texture tiles. If we crunch a whole texture down to its smallest size for fast loading, can we then load individual tiles within that archive? One would assume not. One can't dive into a .zip of a text document and pull out letters indexed at position 15 and 27 without unzipping the whole thing.
     
    BRiT and iroboto like this.
  6. Rikimaru

    Veteran Newcomer

    Joined:
    Mar 18, 2015
    Messages:
    1,040
    Likes Received:
    411
    Data split into 256k chunks (for seek/random access)
    So maximum read overhead for a part of file is 256k - 1byte.



    MS have not said BCPack is lossless, have they?.
     
    #2726 Rikimaru, May 29, 2020
    Last edited: May 29, 2020
    egoless likes this.
  7. iroboto

    iroboto Daft Funk
    Legend Regular Subscriber

    Joined:
    Mar 6, 2014
    Messages:
    12,793
    Likes Received:
    15,308
    Location:
    The North
    Too large for a tile I think. Effective VT solutions should have smaller tiles to reduce bandwidth and footprint in VRAM.

    there are proprietary compressions for textures that will beat block compression and support VT; but I don’t think 256K blocks are the right size. That’s 1/4 MB.
     
    PSman1700 likes this.
  8. senis_kenis

    Newcomer

    Joined:
    Jun 5, 2015
    Messages:
    33
    Likes Received:
    15
    256k chunking is both desirable for seeking/patching and means that all matches in that mode come from a fast local cache, no extra DRAM traffic. Minimizing impact on system mem BW was a design goal.



    What does he mean by fast local cache? is this decompression block local sram?
     
    Lucid_Dreamer likes this.
  9. chris1515

    Legend Regular

    Joined:
    Jul 24, 2005
    Messages:
    6,090
    Likes Received:
    6,327
    Location:
    Barcelona Spain
    Yes the SRAM inside the I/O Complex.

    Other comment are very interesting too.



     
    #2729 chris1515, May 29, 2020
    Last edited: May 29, 2020
    egoless and Lucid_Dreamer like this.
  10. Shifty Geezer

    Shifty Geezer uber-Troll!
    Moderator Legend

    Joined:
    Dec 7, 2004
    Messages:
    43,576
    Likes Received:
    16,033
    Location:
    Under my bridge
    How do you manage data access with irregular compression? 256K of data on SSD will be varying amounts of texture data - one block may contain 10 tiles and another block contain 15.
     
    Lucid_Dreamer, PSman1700 and BRiT like this.
  11. senis_kenis

    Newcomer

    Joined:
    Jun 5, 2015
    Messages:
    33
    Likes Received:
    15
    If Kraken is transparent on system level you will get ssd read (but not ram write) overhead if required data is smaller, he also said "primary mode of use is on data split into 256k chunks", maybe where are other modes?
     
  12. Rikimaru

    Veteran Newcomer

    Joined:
    Mar 18, 2015
    Messages:
    1,040
    Likes Received:
    411
    Theoretically you can request 1 byte and SSD controller reads 256K and only DMA 1 byte to memory.
    I do not think 256K is a lot however.

    Data format can hold offsets and real size of data in a table.
     
  13. Love_In_Rio

    Veteran

    Joined:
    Apr 21, 2004
    Messages:
    1,627
    Likes Received:
    226
    ZX Spectrum looks at you...
     
    HBRU likes this.
  14. Rikimaru

    Veteran Newcomer

    Joined:
    Mar 18, 2015
    Messages:
    1,040
    Likes Received:
    411
    Crash Bandicoot on PS1 paged 64K chunks.
     
  15. manux

    Veteran Regular

    Joined:
    Sep 7, 2002
    Messages:
    2,747
    Likes Received:
    1,907
    Location:
    Earth
    What's the size of pages in sony's ssd? I don't think we know. Often minimum page size on ssd is quite large. Could be the flash pages are 256kB or even 512kB. The optane argument really is valid for small pages as ssd's do have overhead on the minimum read size and optane can read storage like ram. That's also the same reason why trim was invented and early ssd drives could hitch when disk starts to be full(ish). There would be free space on disk but no free blocks. Drive would on writes then have to read many blocks that are not completely full, rearrange data and write back. Trim does this behind the scenes.

    Games would have to pack the data into fairly sized chunks to optimize ssd bandwidth. In best case the ssd block size and kraken compression block sizes are same. I assume this is no problem because most data is large and the very small data you can cache in ram anyway if it turns out to be issue(it's small, pack small data together to larger blocks and cache it in ram as needed...)
     
    #2735 manux, May 29, 2020
    Last edited: May 29, 2020
    BRiT likes this.
  16. Shifty Geezer

    Shifty Geezer uber-Troll!
    Moderator Legend

    Joined:
    Dec 7, 2004
    Messages:
    43,576
    Likes Received:
    16,033
    Location:
    Under my bridge
    That reminds me of Sony talking about address tables something something in their IO patents.
     
  17. iroboto

    iroboto Daft Funk
    Legend Regular Subscriber

    Joined:
    Mar 6, 2014
    Messages:
    12,793
    Likes Received:
    15,308
    Location:
    The North
    I’m not sure. Every VT system is different and is capable of different things and has different restrictions. I really don’t know what is and what isn’t possible; VT systems can get fairly complex and as I understand it supporting and trying to solve and tackle various edge/limit cases.

    It would be ideal that the decompression hardware should be used with VT. And it would appear to be a small miss if it didn’t. But, there as many things we can see, many companies can do compression and VT purely in compute shaders. So I’m not sure if they are using kraken or something else.

    But in non VT scenarios this compression hardware and power is still very effective. Which is everything else basically.

    Some of the actual render programmers here can probably give a proper response; but the rabbit hole is going to be deep I suspect.
     
  18. chris1515

    Legend Regular

    Joined:
    Jul 24, 2005
    Messages:
    6,090
    Likes Received:
    6,327
    Location:
    Barcelona Spain
    Tim sweeney told they use the SSD decompression hardware.
     
  19. Ronaldo8

    Regular Newcomer

    Joined:
    May 18, 2020
    Messages:
    257
    Likes Received:
    302
    A byte addressable SSD? The only serious suggested implementation that I know of is one proposed by Samsung R&D in an IEEE paper in 2018.
     
    DSoup likes this.
  20. DSoup

    DSoup meh
    Legend Veteran Subscriber

    Joined:
    Nov 23, 2007
    Messages:
    14,727
    Likes Received:
    10,874
    Location:
    London, UK
    Byte-addressable storage is already a thing in server-space, but there aren't - or were (and I think it's still the case) - any commercial byte-addressable filesystems so they tend to get utilised just like slow RAM with the inherent advantages / disadvantages that you would expect. I think their use is pretty niche, generally if you need byte-addressable storage you're working in a field where funding is sufficient to shove terabytes of RAM into yours server. In my previous job, some of our servers had 2 petabytes of RAM. And it wasn't enough! :no:
     
Loading...
Thread Status:
Not open for further replies.

Share This Page

  • About Us

    Beyond3D has been around for over a decade and prides itself on being the best place on the web for in-depth, technically-driven discussion and analysis of 3D graphics hardware. If you love pixels and transistors, you've come to the right place!

    Beyond3D is proudly published by GPU Tools Ltd.
Loading...