Shadow casting lights: Why is it such a limitation in games?

[VFX_Veteran]

So we are well into year 2 of the next-generation of games, and I've found myself wondering when we will get to the point where we are seeing real time shadows for all light sources in any given scene. I've never worked on hardware before, but I would love to know why this is such a challenge in games.

I've decided to pick up a book on real-time shadows and run through the gambit and see what I come up with using my Titan X here at home. If anyone can shed some specifics working in the industry, I'd love to hear it.

 

[Pixel]

For the average gamer baked shadows vs real time shadows offers very little visual enhancements for many games and in many environments and scenes in those games. Now I'm not saying all games some games like Doom 3 needed that realtime lighting. But take Forza 5 for example, by having baked in shadows that will give them shader headroom for other more visually spectacular things such as achieving a solid 60fps. There may possibly be ESram gbuffer issues as well.

Also developers are having to pull out more stops to achieve a certain leap in visual fidelity given these consoles have far less expensive hardware than last generation when you factor in inflation:
BOM = bill of materials
Xbox 360 BOM = $525 ($606 in 2013 dollars via factoring in inflation)

sources: http://www.theregister.co.uk/2005/11/24/xbox360_component_breakdown/
http://electronics360.globalspec.com/article/2210/xbox-360-teardown
http://www.alphr.com/news/hardware/80708/isuppli-reckoning-the-xbox-bill-of-materials

PS4 BOM + manufacturing cost is at $381.
Xbox One minus Kinect2 BOM = $396 ($396 in 2013 dollars). Alot less spent on the console internals this time around.

Comparing Xbox 360 adjusted to inflation to Xbox One/PS4 there is a $210 and $225 drop respectively in hardware component cost this generation.

In 2005 Microsoft spend an estimated $247 ($287 with inflation in 2013 dollars) on the CPU/GPU of the Xbox 360 Launch BOM.
Sony spent an estimated $100 on the APU in 2013. Microsoft spend an estimate $110 on their apu. Of course you can factor in MS/Sony got a better deal by going with a single chip solution from one single chip maker.

 

[VFX_Veteran]

Well, let's forget about cost for a minute. I'm wondering more if hardware is just not there yet. I'd rather spend resources on lights having the proper behavior and geometry being pushed through the pipeline (for rounded surfaces) than doing anything else more advanced. To me, this is the basics of getting a more realistic image. I understand things like driving games where you are always outside. But most games have interiors or lamps, torches, flashes from bombs, etc..

Take for example, Batman coming out next week. I can bet a dollar to a dime, all those light sources in gotham city will not cast proper shadows.

 

[AlNom]

Someone may chime in more detail or to correct me, but my cursory understanding was that it's traditionally been too expensive to render the scene from the POV of each light source multiple times (CPU overhead) in addition to the rest of the rendering pipe, and then there's the memory cost on top of that.

Multiple shadow maps was one of the examples used for DX12 parallel rendering, so maybe it'll be more of an option in the future, while scaling down for consoles appropriately (still a depth fill cost + filtering + memory etc.).

On the other hand, I suppose it'd still be extra work to flag the light casters for scalability. *shrug*

 

[Pixel]

I'm saying that these AAA games are specifically developed for the consoles. And now we have consoles with only $110 spend on the main chip, vs $247 or 287 if you wanna factor in inflation. Its not the only reasons but its one of several reasons. This is just another reason you see some developers go with technically less demanding lighting/shading solutions.

Also I think draw call limits play an issue. The low cost choice to go with an apu, which forced low clocked cpu cores, lowered the single thread drawcalls dx11 and opengl 4.* could send. Fable is specifically taking advantage of DX12 and multithreaded drawcalls to allow more light sources. There are several DX12 demos out that show off the improved # of light sources.

Check out the Fable thread.

 

[VFX_Veteran]

I'm saying that these AAA games are specifically developed for the consoles. And now we have consoles with only $110 spend on the main chip, vs $247 or 287 if you wanna factor in inflation. Its not the only reasons but its one of several reasons. This is just another reason you see some developers go with technically less demanding lighting/shading solutions.

Also I think draw call limits play an issue. The low cost choice to go with an apu, which forced low clocked cpu cores, lowered the single thread drawcalls dx11 and opengl 4.* could send. Fable is specifically taking advantage of DX12 and multithreaded drawcalls to allow more light sources. There are several DX12 demos out that show off the improved # of light sources.

Check out the Fable thread.

# of light sources isn't the issue I'm talking about. It's the number of shadows produced by those light sources that I am questioning. So basically you are saying that my Titan X could absolutely render shadows for every light source -- yet it struggles to maintain a consistent 30fps @ 4k for Witcher 3 that doesn't cast shadows for every light source?

And yes, I get your point about draw calls budget. But still.. we are talking about DX12 when this should have been explored way back in DX9.

Lastly, I saw one DX12 demo with a mech unit. Even that only had 1 shadow producing light source. In fact, I read that the ray-tracing shadow architecture only works with 1 light source.

 

[milk]

I'd rather spend resources on lights having the proper behavior and geometry being pushed through the pipeline (for rounded surfaces) than doing anything else more advanced. To me, this is the basics of getting a more realistic image.

You'd change your mind the second you saw how basic such game would look under those severely un-balanced trade-offs.

 

[MJP]

Shadow maps are expensive. For every shadow-casting light you have to generate at least one shadow map, and for every shadow map you need to do the following:

• Determine the set of geometry that's visible to the shadow caster, ideally skipping geo that's fully occluded by other geo
• Generate a command buffer with all of the necessary draw calls, state changes, resource bindings, syncs/flushes, etc.
• Transform and rasterize all of the geometry that's casting shadows
• Wait for all of the draw calls to be finished, with their depths writes written to memory. Depending on the hardware, this may involve flushing caches and/or decompressing the depth buffer.
• Perform any conversion or pre-filtering steps, if you're using something like VSM or ESM.
• Sample the shadow map in your compute shader or pixel shader, possibly with lots of fetches if you're using PCF

This can potentially be a lot of work, especially if you have a lot of shadow-casting geometry with high polygon counts. On top of that, you may also need a lot of GPU memory for storing all of your shadow maps. Ultimately it just doesn't scale up that well, especially compared to how cheap it is to scale up on smaller light sources using a deferred renderer.

 

[Rodéric]

You'd change your mind the second you saw how basic such game would look under those severely un-balanced trade-offs.

Not that imbalanced with tesselation you can improve surfaces and get automatic LoD (unfortunately you still need decimated meshes since you need rather dense meshes to tesselate), and simple geometric lights (sphere, cylinders...) are not that expensive either.
After that you need to use physical measures to get their intensities right and correct tone mapping behind that.
So, no, I don't think it would be nearly as bad as you mention, shadows are critical and GI is important though.

 

[VFX_Veteran]

Shadow maps are expensive. For every shadow-casting light you have to generate at least one shadow map, and for every shadow map you need to do the following:

• Determine the set of geometry that's visible to the shadow caster, ideally skipping geo that's fully occluded by other geo
• Generate a command buffer with all of the necessary draw calls, state changes, resource bindings, syncs/flushes, etc.
• Transform and rasterize all of the geometry that's casting shadows
• Wait for all of the draw calls to be finished, with their depths writes written to memory. Depending on the hardware, this may involve flushing caches and/or decompressing the depth buffer.
• Perform any conversion or pre-filtering steps, if you're using something like VSM or ESM.
• Sample the shadow map in your compute shader or pixel shader, possibly with lots of fetches if you're using PCF

This can potentially be a lot of work, especially if you have a lot of shadow-casting geometry with high polygon counts. On top of that, you may also need a lot of GPU memory for storing all of your shadow maps. Ultimately it just doesn't scale up that well, especially compared to how cheap it is to scale up on smaller light sources using a deferred renderer.

I see. I'm very curious about this because I would like to see how much of a hit it would be to hardware using a high-end graphics board with lots of VRAM (i.e. Titan X). Suggesting that it doesn't scale well leads me to believe we just aren't their yet hardware-wise (especially for consoles). Which is too bad, since this is the most jarring thing I notice in games today that literally takes me out of the game.

 

[Andrew Lauritzen]

Suggesting that it doesn't scale well leads me to believe we just aren't their yet hardware-wise (especially for consoles).

No, I think it's pretty clear that shadow maps are just not scalable enough for ubiquitous visibility computations, GI, etc. The problem is we're in a region where the hardware isn't yet fast enough to switch to something other than shadow maps without sacrificing quality. Volumetric stuff (voxel GI and distance fields - both effectively ray/frustum tracing) currently seem to be the fore-runners in terms of replacements, but it'll be a few years before even the highest end hardware can apply those everywhere, and there will still be tradeoffs in terms of resolution.