John Carmack: Doom 4 will support Partially Resident Textures (Tiled Resources)

The megatextures are huge. I think they'd still need to be compressed for a system with 5GB of available RAM.
You don't need much RAM or BW for the tiles apart from caching them. In the ideal case, you only need as many texels and pixels. Obviously megatexturing isn't 100% efficient, but you have a very low overhead. I'm sure Sebbbi's post on tiled rendering goes into detail and we can put a figure on 1080p requirements. The further storage would be for caching data to address drive concerns, and all the non-tiled-texture data.

The need for megatextured compression is to get the source data to fit on disk and be readable at speed off clunky drives. Once in the console, it's extremely efficient.
 
You don't need much RAM or BW for the tiles apart from caching them. In the ideal case, you only need as many texels and pixels. Obviously megatexturing isn't 100% efficient, but you have a very low overhead. I'm sure Sebbbi's post on tiled rendering goes into detail and we can put a figure on 1080p requirements. The further storage would be for caching data to address drive concerns, and all the non-tiled-texture data.

The need for megatextured compression is to get the source data to fit on disk and be readable at speed off clunky drives. Once in the console, it's extremely efficient.

This one? http://forum.beyond3d.com/showpost.php?p=1580827&postcount=39

Been a while since I read it.

So, 1080p would require about 9216x9216 (arrived at this by 4096x4096 by 2.25 for 1080p as suggested in sebbbi's post). At 2.5 bytes per pixel (from sebbbi's post) you'd be at 212 MB. Seems like the 5400 rpm drives should be fast enough to handle streaming tiles into the tile cache.

edit:
Actually, it's interesting. Assuming a 90 degree, field of view, you'd need four 1080p images to cover a full 360 degrees. Those 5400 rpm drives start to sound kind of slow for filling that tile cache for four distinct views. Sounds to me like you'd want to keep tiles in RAM rather than streaming them off the HDD. You'd need some kind of predictive strategy for doing that.
 
Didn't Rage have a CUDA implementation on Nvidia systems for texture de/recompression?

Because... if the DMAs could be used to transcode the texture data for free, that work could be done by Compute on PS4 "for free" (as in the same game on both consoles 12 vs 18 ALU, 6 left over for free).

http://www.nvidia.com/content/GTC-2010/pdfs/2152_GTC2010.pdf

170-450 MBs of HD Photo compressed textures works out to 4-10 GBs of DXT compressed textures and 20-50 GBs of uncompressed data.

You would only need 10% of the RAM allocated to games in the PS4 and XB1 to store that amount of HD Photo compressed data but the RAM allocation would be too small to hold all that data transcoded into a DXT format.
 
Last edited by a moderator:
This one? http://forum.beyond3d.com/showpost.php?p=1580827&postcount=39

Been a while since I read it.

So, 1080p would require about 9216x9216 (arrived at this by 4096x4096 by 2.25 for 1080p as suggested in sebbbi's post). At 2.5 bytes per pixel (from sebbbi's post) you'd be at 212 MB. Seems like the 5400 rpm drives should be fast enough to handle streaming tiles into the tile cache.

You're doubling both vertical and horizontal resolution, which results in quadrupling the texture resolution. A 2.25 time increase would be 6144*6144 or 90MB!
 
You're doubling both vertical and horizontal resolution, which results in quadrupling the texture resolution. A 2.25 time increase would be 6144*6144 or 90MB!

I couldn't tell which he meant in the post. 90mb definitely makes things a lot more manageable, but you're still at 270mb for a 360 degree view. Not sure how fast the drives are in these consoles.

Edit:
I also assumed 2.5 bytes per pixel, but that may not be the same for the new Doom. I have no idea what it was for Rage. That number is from the material system for Trials Evolution, not even Trials Fusion.
 
You'd certainly be caching some tiles, but that's very predictable in a game with clearly defined movement options. On foot, you need tiles for immediate views and where the player might go over the next half second or whatever. In an aircraft, you need a lot more area covered but at lower LOD either due to height or speed passing over a feature.
 
I know Carmack has had some pretty long talks about how the caching worked on Rage. It would be interesting to go back and listen to it and see what can be extrapolated for the new Doom.


Edit: Rage can consume large amounts of VRAM on PC because of preemptive caching. I'd expect Doom to consume quite a bit of memory for megatexture.
 
Aaaah, that video... it sounds familiar. :p It seems that there are a few ways around it, although I want to see that in games, live.

Here ya go
Thanks for sharing, Davros. I've never heard of that game. It's a start, not as hot as a Doom game, but it seems to work handsomely when composing elements of the image together.

Hardware support for partially resident textures (called Tiled Resources in DirectX API, or more generally "sparse textures"), do not bring any new possibilities over software virtual texturing implementation (custom shader code). It mostly shaves off a few ALU instructions (and one cache friendly memory load), makes anisotropic filtering easier to implement properly and doesn't need tile borders for filtering (that saves less than one percent of storage space, but makes addressing easier because tile borders make power of two alignment hard).

Hardware PRT also provides a minor memory saving (~10 megabytes total), as you don't need your own indirection lookup texture. However if you use a hash instead of a texture with full mip chain (one pixel represents one page, for example 128x128 pixels of texture data), your memory consumption is equal to hardware PRT version. But this requires one extra memory lookup (if cuckoo hash is used) and some extra ALU.

Yes, all modern GPUs support sparse textures. OpenGL 4.4+ and DirectX 11.2+ has support for it on PC.
I am a big proponent of anything hardware, although of course you can match those features with software. Still, in real life term, if you had to choose... would you go with hardware implementation or a software one?

I don't think that's something you really believe you do have to worry about... judging by your work, there can be a few ways around that for you.

Still, trying to put myself in your shoes, if you save ALUs and a little amount of memory, I'd go with the hardware solution.

Could yours and Carmack's previous streaming solutions be considered early implementations of Tiled Resources?

Your work on Trials and what Carmack did with Rage was never heard of before, afaik. Plus Carmack said that something like the Megatexture will win in the end.
 
I have it, how would i tell ?
I do know it uses the same engine
Here is the paper describing how they used it in dragon commander.
http://twvideo01.ubm-us.net/o1/vault/gdceurope2013/Presentations/824810SwenVincke.pdf

In which they used it only for landscape and only 16kx16k texture per level.
Game had ~6GB of textures in 54 levels. (~110MB per level)

If game uses it in same way there should be a folder in install directory with many >~100MB files or some huge files. ;)


Of course it's possible that they have changed to something like Battlefield 3&4 and create the final texture in flight using splatting..
 
One very important issue about Rage was that most of the environment had no dynamic attributes at all, not even specular highlights - because the environment only had one texture layer. This was mostly because of background storage issues though (not being able to fit normal and specular map layers on the DVD) and not a runtime memory concern.

However, Doom 4 will absolutely require dynamic lighting and thus at least two extra RGB layers (normal and spec color) and one or two A channels (specular roughness and perhaps material ID). So we're talking about a significantly larger RAM need - but then the new systems have almost 10 times as much space, too...
 
One very important issue about Rage was that most of the environment had no dynamic attributes at all, not even specular highlights - because the environment only had one texture layer. This was mostly because of background storage issues though (not being able to fit normal and specular map layers on the DVD) and not a runtime memory concern.

However, Doom 4 will absolutely require dynamic lighting and thus at least two extra RGB layers (normal and spec color) and one or two A channels (specular roughness and perhaps material ID). So we're talking about a significantly larger RAM need - but then the new systems have almost 10 times as much space, too...
Yup.
Personally wouldn't mind a space for a displacement map either. (normal or vector.)
Should be a nice fit to mega texturing.
 
One very important issue about Rage was that most of the environment had no dynamic attributes at all, not even specular highlights - because the environment only had one texture layer...
Yep, good point. Some of these won't compress too nicely with JPEG either, I'm thinking, making the streaming demands higher and need for more cached tiles greater.
 
Yep, good point. Some of these won't compress too nicely with JPEG either, I'm thinking, making the streaming demands higher and need for more cached tiles greater.

So with a 90 degree field of view, you're well past 270mb for a full 360 degree view. It'll be interesting to see if Doom has the same pop-in issues with texture streaming that Rage did on the PC.
 
So with a 90 degree field of view, you're well past 270mb for a full 360 degree view. It'll be interesting to see if Doom has the same pop-in issues with texture streaming that Rage did on the PC.

The issues Rage suffered from aren't nearly as pronounced anymore. I just ran through the game with an AMD gpu, so I was doing transcoding on the CPU. I scored just under 100 on the texture benchmark (in 64bit mode).

Wolfenstein is tech5, and is even less noticeable.
 
So with a 90 degree field of view, you're well past 270mb for a full 360 degree view. It'll be interesting to see if Doom has the same pop-in issues with texture streaming that Rage did on the PC.
When using virtual texturing, the memory usage is tightly tied to the rendering resolution. At 720p on Xbox 360, our virtual texturing implementation used a two layer 4096x4096 DXT5 compressed atlas to store all the cached texture data. This is 32 MB. The system also needs an 2048x2048 16 bpp (RGB565) indirection texture (with a full mip chain). This takes an additional 10.6 MB of memory (8MB+2MB+0.5MB+0.125MB+...). So the total memory cost is 42.6 MB at 720p. We aim at 33% working set (33% of the cache is needed to store the texture data for the currently visible frame). This is a good compromise between memory usage and the HDD traffic.

Two DXT5 layers provide albedo.rgb, tangent space normal (xy) in the DXT5 alpha channels, and three additional (low quality entangled) material channels (roughness, grayscale specular multiplier, custom shader specific channel). This was OK for last generation game, but obviously next gen (and PC) games require more shader complexity and better compression quality.

1080p requires 2x cache size compared to 720p (2x = 2.25x pixel count, but with a slightly lower page overhead assuming the same 128x128 page size). On a next gen game you likely also need one extra BC7 compressed layer (1 byte per pixel) to store more material properties at higher quality (and put the color specular multiplier to the rgb triplet of the second DXT5 layer). This is 2x*1.5x = 3x memory for texture data compared to the last generation layout. The indirection texture stays the same (262144 x 262144 virtual texture size is enough). Texture data size is 8192 * 4096 * 3 bytes = 96 MB. Add indirection texture, and you get 96 MB + 10.6 MB = 106.6 MB.

I personally like 8192*8192 cache more than 8192*4096, because modern hardware has extra memory to spare, a bigger cache means less HDD traffic, and the bigger cache handles up to 2560x1600 without any problems. This is 192 MB + 10.6 MB = 202.6 MB. This cache also is big enough to handle a full 360 degree spinning at 1080p without loading any new data. However loading while spinning is not proven to be a big problem. On consoles it's impossible to see any popping in this case either, but on PC with mouse aiming the turning is so much faster that spinning might be a problem for a first person shooter if the texture cache is too small.
 
On PC, the Windows hard drive caching uses your extra RAM to cache the HDD data. Assuming you have 8 GBs or more memory, there will be at least a gigabyte of cached hard drive data available. Because of this, the camera spinning never really accesses the hard drive after the first spin (loading data back is basically just a memcpy operation from the memory cache), even if your own virtual texture cache was too small to contain the full spin.

When the OS caches the HDD data, the data is cached in the final compressed format. This means that it takes less memory, but if the data is required again, it needs to be transcoded again to a GPU filterable texture format. I don't know how Rage does this, but we do limit the number of page loading operations per frame, because the transcoding and the decal blending take both CPU and GPU time. If too many pages are loaded per frame, it might cause a frame rate drop in graphically heavy scenes. The page loading limitation might cause popping on a PC first person shooter game (with a ultra fast mouse look).
 
Last edited by a moderator:
Back
Top