Digital Foundry Article Technical Discussion [2021]

sad watching this remaster remember once Crytek was top tier dev team in aspect of pushing graphics boundries

 

The last impressive game they made was Ryse.

 

hunt also looks quite good

 

I don't believe Sony or Microsoft every published specifications for their custom zlib hardware. Mark Cerny said it works "on the fly" which suggests it's decompresses as quickly as data is pulled off the HDD. The hardware in both PS4 and XBO would have to be utter shite to be outpaced by a Jaguar APU software-based solution.

 

I would've thought it's not that simple.

Maybe for the way it's being compressed it's not as efficient using the hardware.
Maybe dictionary is bigger than hardware supports for the way Kraken is using it.

 

This is where I'd like to see a VRR toggle and not just a more aggressive DRS.
Dropping to 50+ fps isn't a problem for VRR.

 

MS may have not published it. But I believe details were published by VGleaks.

Edit:

Here u go.

https://vgleaks.com/world-exclusive-durangos-move-engines/


Generic lossless compression and decompression
One move engine out of the four supports generic lossless encoding and one move engine supports generic lossless decoding. These operations act as extensions on top of the standard DMA modes. For instance, a title may decode from main RAM directly into a sub-rectangle of a tiled texture in ESRAM.

The canonical use for the LZ decoder is decompression (or transcoding) of data loaded from off-chip from, for instance, the hard drive or the network. The canonical use for the LZ encoder is compression of data destined for off-chip. Conceivably, LZ compression might also be appropriate for data that will remain in RAM but may not be used again for many frames—for instance, low latency audio clips.

The codec employed by the move engines is LZ77, the 1977 version of the Lempel-Ziv (LZ) algorithm for lossless compression. This codec is the same one used in zlib, glib and other standard libraries. The specific standard that the encoder and decoder adhere to is known as RFC1951. In other words, the encoder generates a compliant bit stream according to this standard, and the decoder can decompress certain compliant bit streams, and in particular, any bit stream generated by the encoder.

LZ compression involves a sliding window and operates in blocks. The window represents the history available to pattern-match against. A block denotes a self-contained unit, which can be decoded independently of the rest of the stream. The window size and block size are parameters of the encoder. Larger window and block sizes imply better compression ratios, while smaller sizes require less calculation and working memory. The Durango hardware encoder and decoder can support block sizes up to 4 MB. The encoder uses a window size of 1 KB, and the decoder uses a window size of 4 KB. These facts impose a constraint on offline compressors. In order for the hardware decoder to interpret a compressed bit stream, that bit stream must have been created with a window size no larger than 4 KB and a block size no larger than 4 MB. When compression ratio is more important than performance, developers may instead choose to use a larger window size and decode in software.

The LZ decoder supports a raw throughput of 200 MB/s compressed data. The LZ encoder is designed to support a throughput of 150-200 MB/s for typical texture content. The actual throughput will vary depending on the nature of the data.



Read more https://vgleaks.com/world-exclusive-durangos-move-engines/

 

So, still drops to <60. I don't really think this is a problem of DRS, more a "single-thread-old-gameengine" problem.

 

Sounds about right. Plus, a more modern format like Kraken might offer better performance and a better ratio. Or, cross platform compatibility if you are shipping on PC, mobile, or somewhere else.

 

Lz compression was designed to have one decompression method, it's only the method of compression that varies. So how the data is compressed does not matter, it's not like variable compression or encoding technologies where the complexity of compression impacts decompression. This is why Lz and zlib is so popular. You can be sure that a zlib decompression hardware built now (or ten years ago) will work forever and benefit from improved compression techniques.

 

Logically why would rad advertise oodle for ps4/xbo if it was not competitive with zlib hw in those devices? PC has a different sku so no problem doing oodle there and zlib on last gen console if the performance indicated it. Its just at build time/creation you need to define what to use. And then calling different decompression methods

 

Oodle has nothing to do with zlib. Currently on PS4 and Xbox all data is compressed by zlib which is ideal because both those machines have hardware zlib decompressors than can work on the fly, albeit at a rather reduced pace because they have being first designed to decompress data from the disc.

https://www.gamasutra.com/view/feature/191007/inside_the_playstation_4_with_mark_.php?page=3

 

As mentioned in @dobwal post Digital Foundry Article Technical Discussion [2021]

I wasn't talking about compatibility but efficiency.
In that post you can see the block limit (i referred to it as dictionary size) and also the throughput limit.
Both things that may make using Kraken on cpu a better solution.

 

Using compression/decompression on non-fixed function hardware is probably the way forward. Yes ineffective decompression is a magnitude slower, but newer decomp tech could/is much more flexible and doesnt require the need for fixed function hardware/blocks.
GPU's can be used for decompression aswell, its much faster, more flexible and doesnt require the need for additional die.

 

Not sure I would personally go that far, especially when talking about consoles.

 

I would like to see decompression in amd's next gen io-die. Then forward the uncompressed data over pcie5 bus either to gpu or main ram depending on use case. Should be highly efficient, flexible and low power. Ideally make the decompression block also support zip and other formats and hook it up so that application installs etc. can also benefit from hw decompression. It frustrates me to no end to see those progress bars when installing(decompressing) apps/updates. Microsoft could integrate this into windows so any app using the windows built in decompression would hit fast path if decompression hw is available.

 

From Oodle site http://www.radgametools.com/oodlekraken.htm

I read that as RAD want you to use Oodle Kraken instead of zlib and that it will be competitive, probably/maybe even better than using the zlib HW on the PS4/XBO.
Why would they highlight AMD Jaguar chip in PS4 and XBO if it was "worse" than the builtin stuff?


Going on, if we assume that we are looking at it from a total cost pov, and with even more assumptions.

1. With Kraken you create smaller files
2. With Kraken you decode a file quicker than LZ if the files are the same size on the cpu, but HW LZ is quicker per 1MB by some ratio.

Now if Kraken file is 1MB and the LZ file is 2MB due to #1.
And you decode a 1MB in 8ss on Kraken CPU and 6s on LZ HW (silly numbers, but easy to follow)
So even if the CPU is slower to decode on PS4/XBO the total win in total is still with Kraken.
Of course you need to factor in other resources like cpu time (multi-threaded), bandwidth etc etc but Kraken on CPU can then be better than LZ HW.

 

I'm lost. Efficiency of zlib PS4/XBO hardware compressor vs CPU, or zlib decompressor (hardware/CPU) vs software kraken on PS4/XBO? Or something else?

I can well believe software kraken decompression is faster in the round than software LZ decompression because LZ is ancient. For previous gen consoles I think the key comparison is software kraken decompression versus hardware LZ decompression. And even if software kraken is faster, is there sufficient CPU overhead to shift from hardware LZ decompressions to software kraken decompression?

And to be clear, I don't the know the answer to any of these questions!