Object/texture space lighting

sebbbi

Veteran
Dan Baker's excellent GDC presentation made object/texture space lighting again a popular topic of discussion. Texture space lighting has also been always among my interests, as I have done lots of virtual texturing related research in the past 7 years (we almost beat Rage to ship the first fully virtual textured game).

Dan's excellent presentation:
http://oxidegames.com/wp-content/uploads/2016/03/Object-Space-Lighting-Rev-21.pptx

Jon Greenberg's response:
http://deadvoxels.blogspot.fi/2016/03/everything-old-is-new-again.html

I am not going to repeat all the technical details. I suggest reading both of these links before reading my post any further.

Advantages in texture space processing / lighting:
- Temporally stable. No flickering texture aliasing. (highly important for VR)
- Perfect for data caching: No need to reproject and refilter (advection = blurriness, data validity, etc issues)
- Shading resolution doesn't need to match the final output. Can render OoO (DOF) and motion blurred surfaces with less resolution. As the result is temporally stable, blurring it will look almost identical to full res rendering.
- Update rate doesn't need to match the final output. 30 fps lighting in 90 fps VR game would definitely allow more complex content.
- Foveated rendering (VR) could render surfaces near the focus point at full resolution, and lower the shading resolution elsewhere.
- Sharing shading results between two eyes (VR). Specular needs special care.
- Textures data can be fetched by memory loads instead of texture sampler (never TMU/filter bound).
- Needs only a single anisotropic texture fetch (to fetch the final shaded data). Significantly faster than fetching all texture data with anisotropic filter.
- Can implement own custom data formats / bit packing, as texture data doesn't need to be filtered (*).
- Modern high poly games can lose up to 40% of their pixel shader ALU because of quad inefficiency (small triangles). Texture space lighting has full efficiency always.
- Many others...

Disadvantages in texture space processing / lighting:
-
The amount of shaded texels is rougly 2x-3x higher than shaded screen space pixels. Assuming that we don't shade anything at lower resolution or at lower frame rate.
- If shading the whole object with no sparse paging or occlusion optimizations (see below) the added cost is unbound (lots of large objects near the camera) assuming we want pixel perfect quality.
- Needs unique UV mapping for every surface. Similar to lightmapping, but you don't have to store this data to disc/HDD. You can blit from optimized storage mapping to run time mapping on demand.

Lighting and AO can be both done in texture space (assuming unique UV mapping). But texture space processing also means that you can do decal blending and material masking & combination (such as rust/decay) in texture space and store the results, instead of calculating them again every frame. See my SIGGRAPH presentation for more info.

So lets' begin...


Take a look at slide 61 of the Oxide presentation. It lists four challenges. Virtual texturing + deferred texturing solves three out of four of these challenges. See our SIGGRAPH presentation (link below).

Link to the thread discussing my SIGGRAPH presentation:
https://forum.beyond3d.com/threads/gpu-driven-rendering-siggraph-2015-follow-up.57240

Challenge 1: Dealing with large meshes which have varying density on screen
Virtual texturing trivially solves this. All meshes are split to pages (each containing usually 64x64 or 128x128 pixels). Virtual texture cache (for example a 8196x8196 pixel atlas containing these pages in a grid) contains all the visible texture data. I large mesh close to camera uses more pages, but also covers up as many pixels from the screen as it takes, keeping the total page count roughly the same. Page size is always fixed. Lower mip pages cover larger area of the object. This keeps screen space size of pages roughly the same regardless of the surface's distance from the camera. Virtual texturing has roughly a fixed memory footprint, no matter how complex the scene, or how many big/small objects/textures are used.

Challenge 2: Occlusion system would be needed for an FPS style game
"Virtual deferred G-buffer" stores UV coordinates to the virtual texture cache atlas. You first render this cheap G-buffer. The G-buffer contains UV of each visible surface pixel (occluded pixels are not present). You go through the G-buffer to build various structures (for depth pyramid culling, virtual shadow mapping, etc). During this step, you also identify which pages are resident in the virtual texture cache and which are not. You simply append all the visible page ids to an append buffer, and only shade these pages. This is a huge saving compared to shading the whole texture of an object (at single mip level based on distance). Trivial object space shading is wasteful, as half of the shading results are ignored (surfaces are back facing). Occluded areas are also shaded. Page granularity shading solves these issues.

To further reduce the overshading cost, you should generate a 2d visibility bitmask per each page (for example 64 bits = 8x8 bits = 16x16 texture space tiles = good group size for shading). It is trivial (and practically zero cost) to generate the visibility masks using atomic OR operations when you go through all the UVs of the G-buffer.

Challenge 3: Integrating any screen space techniques
With the "Virtual deferred G-buffer" you have a depth buffer rendered before you light anything. You do lighting (compute dispatches) after you have rendered the G-buffer and identified the required texture pages. You have the possibility to run screen space techniques between these steps. In the texture space lighting shader you simply transform the texel to screen space and lookup the screen space SSAO and SSR values.

You should accumulate the SSAO / SSR to the texture space surface cache instead of reprojecting the screen space data. This guarantees that the history data is always valid, and there is no blurring from repeated filtering (a bicubic advection filter still adds noticeable blur and costs quite a bit -- see here: https://www.shadertoy.com/view/lsG3D1).

You can also use stochastic methods + accumulate the shading in texture space. For example if you want to simulate area lights by cones, you could jitter the direction inside the cone and accumulate the results. This works better compared to screen space techniques (such as temporal AA), for the reasons described above.

Cheap enough?

The baseline cost is 2x-3x of the traditional screen space shading. The quality is better. The question really becomes: Are you able to reuse data (two eyes in VR, lots of similar impostors, etc), use lower resolution rendering in out of focus areas, lower shading update rate (30 fps shading?), choose shading rate based on content: high contrast vs low contrast, highly specular?, crossing a shadow edge?, area covered by a particle effect (smoke, explosion)?, some error metric to check difference? inside screen extension (if rendering extra screen borders for SSAO/SSR)? I am eagerly waiting for someone to combine these techniques together.

The big remaining problem of texture space lighting is the requirement of having unique UV mapping everywhere. Games that have lots of huge objects such as buildings (sky scrapers), bridges (several kilometer long) etc objects that require huge amount of virtual mapping space need to solve this problem somehow.

I recommend reading Ka Chen's excellent GDC presentation from last year (Adaptive Virtual Textures): http://www.gdcvault.com/play/1021761/Adaptive-Virtual-Texture-Rendering-in. A similar solution could be used in solving the unique UV mapping problem.

(*). Custom texture data packing schemes are plausible as filtering is not needed. As data fetch is always 1:1 and shading occurs in 16x16 tiles, can also implement own custom block compression schemes. Cross lane operations are useful for uncompression (assuming threads mapped to pseudo-morton order that groups wave threads as 8x8 tiles).
 
Last edited:
The big remaining problem of texture space lighting is the requirement of having unique UV mapping everywhere. Games that have lots of huge objects such as buildings (sky scrapers), bridges (several kilometer long) etc objects that require huge amount of virtual mapping space need to solve this problem somehow.
Are there even any proposed solutions to this? With my very rudimentary understanding of this it seems like something that cannot be easily (or even not easily) dealt with. Normally when you describe a problem you go on to explain possible solutions :)
 
Last edited:
Are there even any proposed solutions to this? With my very rudimentary understanding of this it seems like something that cannot be easily (or even not easily) dealt with. Normally when you describe a problem you go on to explain possible solutions :)
You don't want to map all objects in advance to a huge virtual address space. Your virtual->physical mapping table (also known as indirection texture) would become huge. You could obviously use hash instead of a flat mapping table. This would give you "infinite" resolution everywhere (assuming you can generate content of any mip, storing unique data would of course be impossible). Hashing on GPU is doable, but whether you opt for closed or open addressing, you have to handle collisions. This means looping, and possibly different loop count among threads of the same wave (= bad). Cuckoo hashing is guaranteed O(1) lookup without loops/branches. But it will cause two cache misses instead of one (assuming good linear probing algorithm, such as robin hood hashing). Per pixel hashing still sounds like a little bit too much for me.

A 10km * 10km terrain requires several million^2 pixels of resolution up close. That's is also too virtual address space much for a flat mapping table. Fortunately for terrain there exists easy solutions (such as clipmaps). Many games have shipped with huge virtual textured terrains. Quake Wars was the first. RedLynx Trials Evolution/Fusion. Battlefield 3 ->, Far Cry 4, etc.

Unique UV mapping (charting) of meshes is not a problem itself. Lightmappers have done this for a long time. You just can't store uniquely charted data for large surfaces that are usually textured with repeating (tiling) textures. The storage cost would be enormous. Id tried this in Rage, and it required 3 DVDs with heavy custom compression, and still the texture resolution was lacking. But there's no technical roadblocks in storing the data to disc/HDD traditionally and expanding to unique mapped surface pixels to virtual texture cache (at runtime). It just needs a "little bit" of work :). My personal experience is that virtual texturing systems tend to become very complex as it is not an isolated rendering component. Virtual texturing is deeply tied to the content management, packaging and streaming (and even game patching/updating).
 
Back
Top