John Carmack: Doom 4 will support Partially Resident Textures (Tiled Resources)

Discussion in 'Console Technology' started by Cyan, Sep 21, 2013.

  1. Davros

    Legend

    Joined:
    Jun 7, 2004
    Messages:
    15,368
    Likes Received:
    2,559
    yes allthough i dont know how much faster it was
     
  2. Shifty Geezer

    Shifty Geezer uber-Troll!
    Moderator Legend

    Joined:
    Dec 7, 2004
    Messages:
    41,727
    Likes Received:
    12,801
    Location:
    Under my bridge
    You don't need much RAM or BW for the tiles apart from caching them. In the ideal case, you only need as many texels and pixels. Obviously megatexturing isn't 100% efficient, but you have a very low overhead. I'm sure Sebbbi's post on tiled rendering goes into detail and we can put a figure on 1080p requirements. The further storage would be for caching data to address drive concerns, and all the non-tiled-texture data.

    The need for megatextured compression is to get the source data to fit on disk and be readable at speed off clunky drives. Once in the console, it's extremely efficient.
     
  3. Scott_Arm

    Legend

    Joined:
    Jun 16, 2004
    Messages:
    13,568
    Likes Received:
    4,210
    This one? http://forum.beyond3d.com/showpost.php?p=1580827&postcount=39

    Been a while since I read it.

    So, 1080p would require about 9216x9216 (arrived at this by 4096x4096 by 2.25 for 1080p as suggested in sebbbi's post). At 2.5 bytes per pixel (from sebbbi's post) you'd be at 212 MB. Seems like the 5400 rpm drives should be fast enough to handle streaming tiles into the tile cache.

    edit:
    Actually, it's interesting. Assuming a 90 degree, field of view, you'd need four 1080p images to cover a full 360 degrees. Those 5400 rpm drives start to sound kind of slow for filling that tile cache for four distinct views. Sounds to me like you'd want to keep tiles in RAM rather than streaming them off the HDD. You'd need some kind of predictive strategy for doing that.
     
  4. dobwal

    Legend Veteran

    Joined:
    Oct 26, 2005
    Messages:
    5,163
    Likes Received:
    1,172
    http://www.nvidia.com/content/GTC-2010/pdfs/2152_GTC2010.pdf

    170-450 MBs of HD Photo compressed textures works out to 4-10 GBs of DXT compressed textures and 20-50 GBs of uncompressed data.

    You would only need 10% of the RAM allocated to games in the PS4 and XB1 to store that amount of HD Photo compressed data but the RAM allocation would be too small to hold all that data transcoded into a DXT format.
     
    #64 dobwal, Jul 22, 2014
    Last edited by a moderator: Jul 22, 2014
  5. TheWretched

    Regular

    Joined:
    Oct 7, 2008
    Messages:
    830
    Likes Received:
    23
    You're doubling both vertical and horizontal resolution, which results in quadrupling the texture resolution. A 2.25 time increase would be 6144*6144 or 90MB!
     
  6. Scott_Arm

    Legend

    Joined:
    Jun 16, 2004
    Messages:
    13,568
    Likes Received:
    4,210
    I couldn't tell which he meant in the post. 90mb definitely makes things a lot more manageable, but you're still at 270mb for a 360 degree view. Not sure how fast the drives are in these consoles.

    Edit:
    I also assumed 2.5 bytes per pixel, but that may not be the same for the new Doom. I have no idea what it was for Rage. That number is from the material system for Trials Evolution, not even Trials Fusion.
     
  7. Shifty Geezer

    Shifty Geezer uber-Troll!
    Moderator Legend

    Joined:
    Dec 7, 2004
    Messages:
    41,727
    Likes Received:
    12,801
    Location:
    Under my bridge
    You'd certainly be caching some tiles, but that's very predictable in a game with clearly defined movement options. On foot, you need tiles for immediate views and where the player might go over the next half second or whatever. In an aircraft, you need a lot more area covered but at lower LOD either due to height or speed passing over a feature.
     
  8. Scott_Arm

    Legend

    Joined:
    Jun 16, 2004
    Messages:
    13,568
    Likes Received:
    4,210
    I know Carmack has had some pretty long talks about how the caching worked on Rage. It would be interesting to go back and listen to it and see what can be extrapolated for the new Doom.



    Edit: Rage can consume large amounts of VRAM on PC because of preemptive caching. I'd expect Doom to consume quite a bit of memory for megatexture.
     
  9. Cyan

    Cyan orange
    Legend Veteran

    Joined:
    Apr 24, 2007
    Messages:
    8,778
    Likes Received:
    2,423
    Aaaah, that video... it sounds familiar. :razz: It seems that there are a few ways around it, although I want to see that in games, live.

    Thanks for sharing, Davros. I've never heard of that game. It's a start, not as hot as a Doom game, but it seems to work handsomely when composing elements of the image together.

    I am a big proponent of anything hardware, although of course you can match those features with software. Still, in real life term, if you had to choose... would you go with hardware implementation or a software one?

    I don't think that's something you really believe you do have to worry about... judging by your work, there can be a few ways around that for you.

    Still, trying to put myself in your shoes, if you save ALUs and a little amount of memory, I'd go with the hardware solution.

    Could yours and Carmack's previous streaming solutions be considered early implementations of Tiled Resources?

    Your work on Trials and what Carmack did with Rage was never heard of before, afaik. Plus Carmack said that something like the Megatexture will win in the end.
     
  10. ToTTenTranz

    Legend Veteran Subscriber

    Joined:
    Jul 7, 2008
    Messages:
    10,502
    Likes Received:
    5,035
    Do you know if the new divinity original sin also uses tiled resources?
     
  11. Davros

    Legend

    Joined:
    Jun 7, 2004
    Messages:
    15,368
    Likes Received:
    2,559
    I have it, how would i tell ?
    I do know it uses the same engine
     
  12. jlippo

    Veteran Regular

    Joined:
    Oct 7, 2004
    Messages:
    1,373
    Likes Received:
    475
    Location:
    Finland
    Here is the paper describing how they used it in dragon commander.
    http://twvideo01.ubm-us.net/o1/vault/gdceurope2013/Presentations/824810SwenVincke.pdf

    In which they used it only for landscape and only 16kx16k texture per level.
    Game had ~6GB of textures in 54 levels. (~110MB per level)

    If game uses it in same way there should be a folder in install directory with many >~100MB files or some huge files. ;)


    Of course it's possible that they have changed to something like Battlefield 3&4 and create the final texture in flight using splatting..
     
  13. Laa-Yosh

    Laa-Yosh I can has custom title?
    Legend Subscriber

    Joined:
    Feb 12, 2002
    Messages:
    9,568
    Likes Received:
    1,454
    Location:
    Budapest, Hungary
    One very important issue about Rage was that most of the environment had no dynamic attributes at all, not even specular highlights - because the environment only had one texture layer. This was mostly because of background storage issues though (not being able to fit normal and specular map layers on the DVD) and not a runtime memory concern.

    However, Doom 4 will absolutely require dynamic lighting and thus at least two extra RGB layers (normal and spec color) and one or two A channels (specular roughness and perhaps material ID). So we're talking about a significantly larger RAM need - but then the new systems have almost 10 times as much space, too...
     
  14. jlippo

    Veteran Regular

    Joined:
    Oct 7, 2004
    Messages:
    1,373
    Likes Received:
    475
    Location:
    Finland
    Yup.
    Personally wouldn't mind a space for a displacement map either. (normal or vector.)
    Should be a nice fit to mega texturing.
     
  15. Shifty Geezer

    Shifty Geezer uber-Troll!
    Moderator Legend

    Joined:
    Dec 7, 2004
    Messages:
    41,727
    Likes Received:
    12,801
    Location:
    Under my bridge
    Yep, good point. Some of these won't compress too nicely with JPEG either, I'm thinking, making the streaming demands higher and need for more cached tiles greater.
     
  16. Scott_Arm

    Legend

    Joined:
    Jun 16, 2004
    Messages:
    13,568
    Likes Received:
    4,210
    So with a 90 degree field of view, you're well past 270mb for a full 360 degree view. It'll be interesting to see if Doom has the same pop-in issues with texture streaming that Rage did on the PC.
     
  17. gurgi

    Regular

    Joined:
    Jul 7, 2003
    Messages:
    605
    Likes Received:
    1
    The issues Rage suffered from aren't nearly as pronounced anymore. I just ran through the game with an AMD gpu, so I was doing transcoding on the CPU. I scored just under 100 on the texture benchmark (in 64bit mode).

    Wolfenstein is tech5, and is even less noticeable.
     
  18. sebbbi

    Veteran

    Joined:
    Nov 14, 2007
    Messages:
    2,924
    Likes Received:
    5,293
    Location:
    Helsinki, Finland
    When using virtual texturing, the memory usage is tightly tied to the rendering resolution. At 720p on Xbox 360, our virtual texturing implementation used a two layer 4096x4096 DXT5 compressed atlas to store all the cached texture data. This is 32 MB. The system also needs an 2048x2048 16 bpp (RGB565) indirection texture (with a full mip chain). This takes an additional 10.6 MB of memory (8MB+2MB+0.5MB+0.125MB+...). So the total memory cost is 42.6 MB at 720p. We aim at 33% working set (33% of the cache is needed to store the texture data for the currently visible frame). This is a good compromise between memory usage and the HDD traffic.

    Two DXT5 layers provide albedo.rgb, tangent space normal (xy) in the DXT5 alpha channels, and three additional (low quality entangled) material channels (roughness, grayscale specular multiplier, custom shader specific channel). This was OK for last generation game, but obviously next gen (and PC) games require more shader complexity and better compression quality.

    1080p requires 2x cache size compared to 720p (2x = 2.25x pixel count, but with a slightly lower page overhead assuming the same 128x128 page size). On a next gen game you likely also need one extra BC7 compressed layer (1 byte per pixel) to store more material properties at higher quality (and put the color specular multiplier to the rgb triplet of the second DXT5 layer). This is 2x*1.5x = 3x memory for texture data compared to the last generation layout. The indirection texture stays the same (262144 x 262144 virtual texture size is enough). Texture data size is 8192 * 4096 * 3 bytes = 96 MB. Add indirection texture, and you get 96 MB + 10.6 MB = 106.6 MB.

    I personally like 8192*8192 cache more than 8192*4096, because modern hardware has extra memory to spare, a bigger cache means less HDD traffic, and the bigger cache handles up to 2560x1600 without any problems. This is 192 MB + 10.6 MB = 202.6 MB. This cache also is big enough to handle a full 360 degree spinning at 1080p without loading any new data. However loading while spinning is not proven to be a big problem. On consoles it's impossible to see any popping in this case either, but on PC with mouse aiming the turning is so much faster that spinning might be a problem for a first person shooter if the texture cache is too small.
     
  19. sebbbi

    Veteran

    Joined:
    Nov 14, 2007
    Messages:
    2,924
    Likes Received:
    5,293
    Location:
    Helsinki, Finland
    On PC, the Windows hard drive caching uses your extra RAM to cache the HDD data. Assuming you have 8 GBs or more memory, there will be at least a gigabyte of cached hard drive data available. Because of this, the camera spinning never really accesses the hard drive after the first spin (loading data back is basically just a memcpy operation from the memory cache), even if your own virtual texture cache was too small to contain the full spin.

    When the OS caches the HDD data, the data is cached in the final compressed format. This means that it takes less memory, but if the data is required again, it needs to be transcoded again to a GPU filterable texture format. I don't know how Rage does this, but we do limit the number of page loading operations per frame, because the transcoding and the decal blending take both CPU and GPU time. If too many pages are loaded per frame, it might cause a frame rate drop in graphically heavy scenes. The page loading limitation might cause popping on a PC first person shooter game (with a ultra fast mouse look).
     
    #79 sebbbi, Aug 9, 2014
    Last edited by a moderator: Aug 9, 2014
  20. Scott_Arm

    Legend

    Joined:
    Jun 16, 2004
    Messages:
    13,568
    Likes Received:
    4,210
    Thanks for correcting my math, sebbbi. Another great post.
     
Loading...

Share This Page

  • About Us

    Beyond3D has been around for over a decade and prides itself on being the best place on the web for in-depth, technically-driven discussion and analysis of 3D graphics hardware. If you love pixels and transistors, you've come to the right place!

    Beyond3D is proudly published by GPU Tools Ltd.
Loading...