Exponential shadow maps, useful?

[sergi.gonzalez]

Hi all,

I've recently tested Marco Salvi's Exponential Shadow Maps using the sample code from ShaderX6 and I'm very disappointed with the results (no offense). The key points are (+pro, -con):

+ minimized self-shadowing artifacts
+ blurred high detail shadows (a pro from my point of view)
- a lot of artifacts (false shadows) appear if the shadow receiver is near to the light
- slope and bias parameters are hard to use

While I was googling information about this tecnique I've found another article ( http://www.cs.ucl.ac.uk/staff/J.Kautz/publications/esm_gi08.pdf) with the same name written by several authors (Annen, Peters, Seidel, etc) that are doing a similar thing. The results are promising but they switch back to PCF in special cases.

What do you think about Exponential Shadow Maps? Have you got good results?

Regards,

s.

 

[nAo]

The editor made a mistake and included in the DVD some old sample code, I will post on my blog the code that was meant to be published.

Regarding the false shadows you are observing as long as you don't run of out range (which might easily happen if you don't use log-filtering) is mathematically impossible to get false shadows as Markov's inequality always under-estimate occlusion (i.e. you get light bleeding)

What do you mean by "slope and bias" parameters are hard to use? Can you elaborate a bit more on this?

 

[sergi.gonzalez]

Regarding the false shadows you are observing as long as you don't run of out range (which might easily happen if you don't use log-filtering) is mathematically impossible to get false shadows as Markov's inequality always under-estimate occlusion (i.e. you get light bleeding).

I uploaded an image http://rs259.rapidshare.com/files/104863976/esm_bleeding.gif showing the artifacts (it's light bleeding indeed).

Ok, when will you post the corrected sample code?

What do you mean by "slope and bias" parameters are hard to use? Can you elaborate a bit more on this?

I was a long time trying to remove the artifacts unsuccessfully.

BTW, did you read Annen's paper?

 

[nAo]

I uploaded an image http://rs259.rapidshare.com/files/104863976/esm_bleeding.gif showing the artifacts (it's light bleeding indeed).

No it's not, they are just self shadowing artifacts due to poor bias parameters.
As I said the code that was put on the DVD is some extremely old code (probably written a year ago) that I used to prototype my stuff. You can even see ringing artifacts due to bad filtering code.

Ok, when will you post the corrected sample code?

Soon, I'm writing a mini paper about ESM (containing some additional insights..) that I will post with the proper code on my blog.

I was a long time trying to remove the artifacts unsuccessfully.

I'm not surprised

BTW, did you read Annen's paper?

Yes, I did.

 

[nAo]

Ok, when will you post the corrected sample code?

Posted a few days ago:
A conceptually simple(r) way to derive exponential shadow maps + sample code

 

[sergi.gonzalez]

Ok, thanks. I'll take a look.

 

[sebbbi]

I just implemented exponential shadow maps to our forthcoming console game, and the technique is working better than I expected. I would like to thank Marco Salvi about this good idea.

The best thing about ESM is that it fits our target platform really well (bandwidth free MSAA and no hardware PCF for depth textures). I dropped our shadow map rendering resolution by 2x2 and enabled 4xMSAA on shadow map rendering when I switched from (4 x point sampled) PCF to single sample bilinear ESM. The shadow map rendering performance got around 3 times faster (four times faster fill and free 4xMSAA), and the shadow map sampling perf hit during material rendering dropped nicely (single bilinear fetch per light + exp instruction is considerably faster than four point filtered PCF samples per light + bilinear interpolation in pixel shader). As we have lots of dynamic lights, our total frame rendering time halved because of this change (16ms -> 8ms). The shadow quality is also slightly better. The bit softer shadows look more natural and all our shadow map bias issues are gone, but the 4xMSAA is not able to completely repair the detail lost by the 2x2 shadow map resolution decrease and the ESM light bleeding reduces the contact shadow quality.

The only minor flaw I found in ESM was the problematic darkening multiplier adjustment. With a small darkening multiplier we get a lot of light bleeding when shadow casters are near the receivers and with a large darkening multiplier the shadows start to look too sharp (much like basic single bilinear hardware PCF when 4xMSAA is active on shadow maps and almost like point sampling when no MSAA was used). The optimal value we are currently using has a little bit of both issues, but on average it looks very good. On games with less depth complexity the techinique would work even better.

Currently I do not do any blurring in addition to the 4xMSAA resolve in the ESM shadowmaps, as it is enough for our purposes (for now). But if a fast blur shows considerable improvement in the image quality, it would be a good addition later on.

 

[sergi.gonzalez]

Esm + Pssm

sebbi: Thanks for the info.

Have you combined ESM with parallel split shadow maps? We are testing ESM with PSSM and we are having a lot of problems trying to get rid of the artifacts (the depth complexity is huge in our scenes).

I realized I don't still thank to thank Marco Salvi for sharing his technique with the community. I take this chance to do it. Great work Marco!

Regards,

s.

 

[sebbbi]

Have you combined ESM with parallel split shadow maps? We are testing ESM with PSSM and we are having a lot of problems trying to get rid of the artifacts (the depth complexity is huge in our scenes).

We have ESM filtering on both our directional sun light (3 split PSSM) and our local (limited range) spotlights. For the spotlights I implemented a extra bit of code to change the exponent multiplier depending on the surface range from the light source. This helped a lot to get the spots working well. For the directional sunlight the bleeding artifacts are considerably more visible for near floor objects and the shadows are much more sharp and aliased for the ceiling. In real life it should be exactly the opposite: the shadows from far away geometry should be smoother and the shadows from near geometry should be sharper. ESM has it's flaws, but compared to 2x2 PCF it looks better in our case at least, and saves a lot of performance (mostly thanks to the free 4xMSAA on shadowmaps).

Blurring the shadow maps should help a lot, since it both makes the shadows smoother and also allows you to increase the exponent multiplier (reducing the bleeding). However for large shadow maps this costs a lot of extra performance.

 

[steel_3d]

Sorry to dig up this post from the dead or step on any toes, but I thought I'd share my experiences with implementing exponential shadow maps in a real game. Hopefully it'll offer some guidance to future adopters.

If you want a shadowing technique that gives soft penumbras AND resolves fine self-shadowing, then exponential is NOT for you. There's a fundamental tradeoff in penumbra softness and depth range/self-shadowing distance. I've tried to illustrate this with my clumsy drawing below:

So as you can see, the closer self-shadowing you're trying to resolve and the larger the depth range in your scene, the less penumbra you get from far-away casters, to the point where shadows from far away casters are essentially sharp edged, as sebbbi has also mentioned. The pixelated look of hardware pcf shadow maps is replaced with a more curvy bilinear filtered edge, but in my opinion it's not worth it. This could be made to work for small depth range lights (point lights, spot lights) like sebbbi says, but I don't think it's adequate for the sun.

On top of this there are a few more issues that come into play if you try to implement this on PS3. The precision of a 16bit buffer is not enough to represent a decent depth range, and you even have to give up half precision computations in pertinent sections of your shader to avoid artifacts. Going to 32bit you also lose hardware bilinear filtering which is a big performance hit. Then there's the blur, which is expensive for my taste for any reasonable sized shadow map, even a separable 3x3 blur. Note that on powerful hardware you could do a really wide blur and trade off overly soft shadows cast by near casters for not-so-hard shadows cast by far casters. You end up throwing away most of your blur from far away casters, as I show in my illustration, because the values grow so fast that they saturate out to "fully in shadow" very quickly.

All in all, I've decided that this technique is not for me, especially on a 60hz game. I could've lived with the performance hit if I was able to use the 16bit zbuffer, but as soon as I tried to resolve shadows closer than 1 foot/avoid light bleeding, the precision ran out. I'm back to sharp high resolution shadows instead. They're faster and don't exhibit light bleeding bugs like all the soft shadow techniques (be it variance, exponential, stochastic).

That said, I commend Marco on a cool addition to the field of real time shadows. It can have its uses in certain applications. I'm looking forward to seeing an implementation in a shipped game.

As a side-rant, I think all papers on shadow techniques should use a full scale world to demonstrate feasibility and accuracy. I'm a little tired of tea pot screen shots

 

[Andrew Lauritzen]

If you want a shadowing technique that gives soft penumbras AND resolves fine self-shadowing, then exponential is NOT for you. There's a fundamental tradeoff in penumbra softness and depth range/self-shadowing distance.

Yup, this is an important point to note for all constant-space/time shadow filtering algorithms. The high-level point is that the situations that you demonstrate there are the ones that become ambiguous ("lossy compression") - i.e. you don't have enough data about the real distribution over the given filter and thus you're going to get it wrong somehow... the different techniques just differ in how and which case they get wrong.

Obviously for hard-edged shadows (pretty small filter widths) most of them do pretty well though, and all do better than PCF in that they can readily handle mipmapping and anisotropic filtering, as well as cheap blurring... but therein lies the most important detail: you'd better have sufficient hardware support for doing that filtering on a high enough precision texture to represent a reasonable depth buffer (i.e. probably 32-bit float in practice for most of these techniques). CSMs and to some extent LVSMs are an exception to this, but you tradeoff doing many low-precision lookups for the single high-precision one.

So I'd agree that these techniques are barely suitable for current-generation consoles - if at all - but on DX10+ class graphics cards you can get both higher quality and even higher performance shadows than with PCF in a lot of cases, and also avoid a lot of the PCF issues with biasing and poor scaling to boot.

That said, I commend Marco on a cool addition to the field of real time shadows. It can have its uses in certain applications. I'm looking forward to seeing an implementation in a shipped game.

I believe Trials HD for Xbox 360 uses ESMs to good effect, but sebbbi can correct me if I'm wrong As far as the other shadowi filtering techniques go, I can think of a number of applications that use standard VSMs (Little Big Planet being the one that comes to mind on PS3).

As a side-rant, I think all papers on shadow techniques should use a full scale world to demonstrate feasibility and accuracy. I'm a little tired of tea pot screen shots

The researchers totally agree with you, but are you willing to donate the art assets for a full scale world from your game into the public domain for them to use? Having worked on the academic side in the past, that's a constant problem there... game developers are rarely willing to share art assets, and thus you use whatever you can get your hands on.

 

[furore]

Here's my experience with ESMs on a nearly out the door XBLA/PSN title:

1. The game is not photo-real, and ESMs can be filtered down to give smooth (but not "soft") shadows. This helped the shadows match the overall art style - the filtering step gives a certain amount of scope for artistic interpretation/interference.

2. The encoding hides acne-type artifacts really well. This is sort of 1. again. When I first got a rough version of the shadowing working with PCF, we thought there was no chance of getting a usable result and were working out how to break the bad news to art. Switching to an ESM encoding really made a big difference.

2a. When the artifacts do show through, they are less discreet than PCF.

3. Using FP32 formats can be painful on consoles, both to write and read. We're almost but not quite carrying the cost on all platforms in a not especially complex 60Hz title.

4. We ended up with nearly no character self-shadows but reasonably soft edges. Happily, this was just what the artists ordered, and the occluder-receiver distance in the environment is generally enough to get the environment casting on itself.


I can see why ESMs worked well for Trials HD (which looks lush). If our camera were just a bit closer to the action, I might be getting away with it. As it is, the smoke and mirrors aren't quite holding up in all cases. We could have used two maps PSM style to get around this, but that would have exacerbated the FP32 cost.


Overall - I would consider using ESMs again, especially in a stylised aesthetic, but not for dynamic environment shadows. I think you get a lot of bang for your pixels, but those pixels can be quite expensive to maintain. It's probably best for casting localized maps, where you prefer smooth edges over fine self-shadowing.


(Usual personal opinion, not representative of employer and especially not publisher caveat applies.)

 

[steel_3d]

sebbbi: How did you end up doing the roof shadows in Trials? Fake secondary shadow texture? The ESM looks good. You can find the light leaking if you're looking for it but I'm sure the average user will have no clue anything's wrong. It's harder to pull off in outdoor scenes with higher contrast shadows.

I'm not sure why people are having issues with self-shadowing artifacts with PCF.

-For solid objects you only render faces facing away from the light, so they don't fight with the lit side.
-You make sure the shadows only modulate the light source that's shining from the shadow direction, otherwise you'll see artifacts on the back side if you're also modulating lights that light the back side (not to mention that it's wrong).
-For open manifolds, alpha tested geometry and such, you do need to render with culling disabled and you will get acne. This can be solved to a satisfactory degree by a per-object bias modulated by slope (with respect to light direction) and some sort of texel size heuristic. Even for trapezoidal shadow mapping for which there's hardly a trivial texel size heuristic I just use a distance + slope based metric and it's good enough. You trade off shadow accuracy for less acne.
-I should note that I do my depth bias at shadow sampling time rather than shadow rendering time. (Hmmm it just occurred to me that it might make more sense to do it at shadow rendering time. I haven't thought about it much but it seems like I could work out a better per-texel depth bias in shadow map space. Is it just max( ddx(light_space_z), ddy(light_space_z) )??).

 

[Andrew Lauritzen]

-I should note that I do my depth bias at shadow sampling time rather than shadow rendering time. (Hmmm it just occurred to me that it might make more sense to do it at shadow rendering time. I haven't thought about it much but it seems like I could work out a better per-texel depth bias in shadow map space. Is it just max( ddx(light_space_z), ddy(light_space_z) )??).

You actually want to do it at sampling time because you want to use a different bias for each PCF sample (if you're doing edge softening) based on where a planar receiver would be at that offset in light space. There's a good explanation in an AMD presentation a while back (can't remember which one but it's on their web site) and there's some code in the cascaded shadow maps example in the DXSDK.

PCF biasing can become a big problem with larger filter kernels though... you can easily get cases with both types of biasing problems (too little and too much) that cannot be solved by any amount of tweaking. Furthermore large filter kernels are expensive with PCF, making "proper" filtering impractical. Thus you'll always get severe aliasing in the distance or when a shadow is projected on a surface that is nearly parallel to the view direction (often the ground or the road unfortunately).

 

[sebbbi]

On top of this there are a few more issues that come into play if you try to implement this on PS3. The precision of a 16bit buffer is not enough to represent a decent depth range, and you even have to give up half precision computations in pertinent sections of your shader to avoid artifacts. Going to 32bit you also lose hardware bilinear filtering which is a big performance hit. Then there's the blur, which is expensive for my taste for any reasonable sized shadow map, even a separable 3x3 blur. Note that on powerful hardware you could do a really wide blur and trade off overly soft shadows cast by near casters for not-so-hard shadows cast by far casters. You end up throwing away most of your blur from far away casters, as I show in my illustration, because the values grow so fast that they saturate out to "fully in shadow" very quickly.

16 bit integer buffer was enough for my purposes, and it provides full speed hardware filtering. The Xbox antialiasing hardware was also really helpful, as it's completely bandwidth free. I used 4x hardware AA for all our shadow maps (to provide us free blur). For all spotlights the antialiasing provided enough blur (spotlights have limited range), but for the CSM sunlight I used a cheap blur filter (four 16 bit integer texels can be filtered and averaged together with one bilinear fetch at half/half texel offset).

We had limited view range, but the factory halls were very tall, so I had to render the CSMs from high distance up from the camera (to make all the ceiling structures and windows in the roof visible in the shadow). As I knew that all the occluders above the area of the camera view frustum were not visible I didn't need to give them any depth range in the map. The min/max depth values are calculated for each cascade (as seen from the sunlight) using the cascade frustum 8 vertices. Depth values before this range can be clamped to zero, and surfaces beyond this range can be discarded completely (viewport culling). The area between cascade min/max vertices is scaled to fill the whole 16 bit integer range. This was really good setup for us, since the camera and the sunlight are almost perpendicular in the game (as it's a 2.5d sidescroller).

sebbbi: How did you end up doing the roof shadows in Trials? Fake secondary shadow texture? The ESM looks good. You can find the light leaking if you're looking for it but I'm sure the average user will have no clue anything's wrong. It's harder to pull off in outdoor scenes with higher contrast shadows.

Opaque roof structure shadows are the same as everything else. However the transparent window shadows, fences, glass objects, etc are rendered to the second channel of G16R16 texture (cleared to 1.0 initially). The transparent shadow casters are rendered after all opaque casters are rendered to the shadow map. Depth read is active (no writes) and I use multiply blend. This way all transparent casters beyond a opaque surface are clipped (these do not affect any pixels). Transparent casters before the opaque surface get multiply blended together, and darken the surface color correctly. The rendering order of the transparent casters do not matter as multiply blend is order independent (no need to sort the casters). When sampling the G16R16 shadow map, the result of the ESM test and the transparent caster channel is multiplied together (we actually subtract the 1-results together to hide edge leaking). This technique provides proper shadows from transparent shadow casters. The only limitation is that transparent caster cannot itself receive transparent shadows from other transparent casters (but it can still receive ESM soft shadows from opaque casters). This was not a limitation for us, since we do not have that many transparent shadow casters (and the windows themselves rarely receive transparent shadows from other objects).

I can see why ESMs worked well for Trials HD (which looks lush). If our camera were just a bit closer to the action, I might be getting away with it. As it is, the smoke and mirrors aren't quite holding up in all cases. We could have used two maps PSM style to get around this, but that would have exacerbated the FP32 cost.

It worked well for us, but I still miss the proper contact shadows we had in Trials 2 Second Edition (PC game that had pixel perfect self shadowing with stencil shadows). In our next game we have to experiment with various SSAO and other ambient occlusion systems to see if the added contact shadows can be used to combat the ESM limitations.

 

[steel_3d]

You actually want to do it at sampling time because you want to use a different bias for each PCF sample (if you're doing edge softening)

I'm not doing edge softening, just using the hardware PCF, but if your filter kernel is constant size in shadow map texels then my idea should still work in theory. If you're measuring the derivatives of Z as you're rendering into the shadow map, you know how much Z is changing over a given shadow map texel (ok I see this can be a bit of an issue with larger kernels unless you assume continuous planar casters over the entire kernel). Now if you bias that shadow map texel's z back by this delta, that surface can't shadow itself. Of course you can still get an unacceptably large bias. I haven't tried this approach yet but I will when I get a chance. It's exact, unlike the distance+slope heuristic I'm using now.

The min/max depth values are calculated for each cascade (as seen from the sunlight) using the cascade frustum 8 vertices.

I see, so the reason you're able to get such soft shadows off the roof is because they're in a different cascade and thus can expose a blur in a different depth range than the ground objects. That's pretty cool, it ends up being a similar approach to Andrew's LVSM idea.

 

[Andrew Lauritzen]

I'm not doing edge softening, just using the hardware PCF

If you're doing more than one hardware PCF tap then you're doing edge softening. If not, then you're not really doing much filtering at all anyways and you probably won't have many biasing problems.

... but if your filter kernel is constant size in shadow map texels then my idea should still work in theory. If you're measuring the derivatives of Z as you're rendering into the shadow map, you know how much Z is changing over a given shadow map texel (ok I see this can be a bit of an issue with larger kernels unless you assume continuous planar casters over the entire kernel). Now if you bias that shadow map texel's z back by this delta, that surface can't shadow itself. Of course you can still get an unacceptably large bias.

That's precisely equivalent to the slope-scale-bias approach (in depth/stencil state). The problem is that for larger filter kernels your filter will cover several receivers and you want to be testing bias against where your *receiver* was in the shadow map, not where your occluder is if it existed at a given pixel... it's hard to explain without pictures, but you always want to be testing what occluder actually covered a sample in your shadow map (not where some theoretical other object might be if it covered there) vs. where your receiver plane is. Obviously high curvature receivers + large filters still causes trouble, but that's unavoidable without a better modelling of an area light or some probabilistic cleverness.

I see, so the reason you're able to get such soft shadows off the roof is because they're in a different cascade and thus can expose a blur in a different depth range than the ground objects. That's pretty cool, it ends up being a similar approach to Andrew's LVSM idea.

Yep always clamp your depth ranges to your actual cascades (or frustum). Furthermore if you need better depth precision (for ESM exponents or otherwise) you can apply the layered idea to any technique... just partition your depth range into N parts and into each part clamp/rescale the range to [0, 1] and filter each independently. It's brute force, but it helps sometimes. For instance, ESM with 4 depth ranges does a pretty decent job of maintaining contact shadows even with c ~88 (no log trickery). That said, for four 32-bit values you may be better off with EVSM.

 

[Brejlounek]

I was experimenting with ESM before and found it quite useful. It worked great with small resolution and 2x2 box blur which blurred even and odd pixels separately.

There were only two downsides: first being bias issues near occluder/caster contact, which can be decreased by shortening the range.

Second one was ugly and already described here by steel_3d, edge softness inconsistency over large depths (even quite small ones). This happnes because the difference between depths is too large - so what if I decreased it? Everytime I'd blur the shadow map, I'd look at values I am blurring, keep the highest one and shift the deepest one to some maximum constant distance from the highest one. That should create some shallower shadow map edges and soften the shadow edge.

Did anyone try that? I didn't yet, it's just an idea I'll definitely try sometime. Sorry to bring up older topic, I came here through google...

 

[Rodéric]

Sorry to bring up older topic, I came here through google...

I don't think it's a problem as all as long as you are contributing to the discussion in a meaningful way (which you are.)