My Stencil Shadow Sample and related problems

[KPixel]

Hello,

I want to show you a sample of my (just started) engine 'ExKalibur' : http://esotech.free.fr/pub/ExKalibur_v0.18beta.zip
edit : new version : http://esotech.free.fr/pub/ExKalibur_v0.19.zip
It feature a Stencil Shadow rendering path (on a 3DS model).
I use the depth pass approach with the constraint "Stencil!=0 => Pixel is shadowed".
The algo is far to the perfection; I have many subtle problems with it :


1- If a model isn't correctly closed (see the Teapot in the sample), I don't get a correct shadow. I know I should fix (at the loading) that type of models, but I haven't had time for that (and it is a low priority TODO )

2- Circles have very jagged shadows. I'm not sure if it is a concequence of the Stencil Shadow method...

3- Boxes's shadows change brutally when the light go from one face to another. I think it is 'normal' (due to the algo used to define lighted/shadowed faces). What is less normal, is the fact that those shadows are progressively distorted four times per round (one per face).
I can't find why...

4- Shadowed areas aren't always correctly shadowed

There is probably others visual bugs and they are certainly linked...


If you can help me...

Hum, on side note, all open source samples (and most docs) I have found on Stencil Shadow use OpenGL (and I use DirectX 9). It really doesn't help

 

[Basic]

This is meant to be contructive critisism, I hope it doesn't sound as anything else.


First, the controls are far too sensitive. A fast tap on the forward key is enough to pass the whole scene for me. (Is it a fixed movement per frame?)

The light isn't realy stopped when pressing 'L'. Because when 'L' is pressed again, the light will jump to the place it would have been at if you hadn't stoped it. (Just minor anoyance, but it makes it harder to examine what happens when the light moves.)

Is there a generic way to close models? You could demand that all models are closed, and maybe add a function that check closedness (that's easy). Objects that aren't closed result in a warning, and should be sent back to the artist.

Even with "standard" lightning, the lightning on the sphere and teapot is rather jagged, and jumps forward as the light moves around. (The jumps can be seen on all objects.)
It seems as the lightning from the moving light is cut of completely when the light is on the wrong side of the vertex normal. (That's OK.) But make sure that the lightning that is cut of already is zero.

On the cuboids there's another (rather common) error. Vertex normals are OK when approximating a smooth surface. But when you have sharp edges, you realy should have one normal for each side of the edge.

The above changes should make object made of flat surfaces look OK with stencil shadows. But it's not enough for the sphere, because the vertex normals don't match the triangle normals. You could have a triangle where some vertex normals are pointing to the light, while the triangle normal points away, and then you'd get a hard lightning border.
I don't know how to remove that most efficiently. But if you make sure that the lightning function fades to zero with some margin as the light vector becomes perpendicular to the normal, and that margin is larger than the difference between the vertex normals and the triangle normals, then it should look OK.
(That's if the stencil shadow operations are OK, so there's some left.)

On to the stencils:
First, it seems as shadows are only cast on surfaces that are directed away from the camera, and then the shadow is seen through the same object. Have you rendered the back faces to the z-buffer before the stencil calculation?

When watching the "Possible silhouette" mode, the extruded edges goes in wrong direction. It looks like each vertex on the object is matched to wrong extruded vertex (the next vertex clockwise on the silhouette).

 

[KPixel]

This is meant to be contructive critisism, I hope it doesn't sound as anything else.

It sound as a constructive critisism; Thanks

The controls are far too sensitive. A fast tap on the forward key is enough to pass the whole scene for me. (Is it a fixed movement per frame?)

Yes, it is I'll fix this...

The light isn't realy stopped when pressing 'L'. [...]

I use the time to compute the position of the light; but I can also correct this one.

Is there a generic way to close models? [...]

Yes. As I said, it is simple a matters of time and priority...

But make sure that the lightning that is cut of already is zero. [...] You realy should have one normal for each side of the edge. [...] But if you make sure that the lightning function fades to zero with some margin as the light vector becomes perpendicular to the normal, and that margin is larger than the difference between the vertex normals and the triangle normals, then it should look OK. [...]

Hum... I can't see how to translate all this in C++ language :?

It seems as shadows are only cast on surfaces that are directed away from the camera, and then the shadow is seen through the same object. Have you rendered the back faces to the z-buffer before the stencil calculation?

I have a first pass for the ambiant/emissive light; so the z-buffer should be correctly filled...

When watching the "Possible silhouette" mode, the extruded edges goes in wrong direction. It looks like each vertex on the object is matched to wrong extruded vertex (the next vertex clockwise on the silhouette).

I'll check if I can see that...

 

[Basic]

But make sure that the lightning that is cut of already is zero.

What is your lightning function? Maybe you're cutting out the ambient part when the light is on the wrong side of the normal.

You realy should have one normal for each side of the edge.

It's not as much describing in C++ as how the modelling is done. Make three vertex+normal pairs for each corner of the quboid, the vertex part should be identical, but the normal part should differ. Then render it with a triangle list.
If you want to do it with a triangle strip, you need to insert degenerate triangles to pass the sharp edges.

...lightning function fades to zero with some margin...

I must admit that I don't know the best way to do it, but here's one try:
Your lightning function likely depends on the vertex normal dot vertex-light-vector and/or the half angle vector. If the dot product is X, then you could add some margin by doing (X-eps)/(1-eps). Maybe that's only neccesary to the diffuse part of the lightning, since the specular part usually already is rather small when the dot product is small.

Have you rendered the back faces to the z-buffer before the stencil calculation?

Move slightly to the right from original position, view stencil shadows and possible silhouette, and stop the light when the shadow from the sphere falls on the big pink cuboid.
You shouldn't be able to see that shadow, but it's still visible. And the shape is just right for a shadow that falls on the three distant sides of the quboid. You can also see how the "edge rays" from the sphere passes through the quboid. (It can be seen on other places too, but thats the easiest to spot.)

looks like each vertex on the object is matched to wrong extruded vertex

It's easiest to see in "possible silhouette" + wireframe mode, when the "edge rays" are pointing away from you.


Heh, five quotes and all from me, how's that for a narcissistic post.

 

[KPixel]

Thanks Basic I think all I need now is time...

Ciao.

 

[NeoCool]

The link doesn't seem to work for me...any ideas?

 

[KPixel]

The link doesn't seem to work for me...any ideas?

Well, I don't know... It works for me. Try again, the server was probably down.

 

[LeGreg]

more polygons wouldn't hurt especially
if you try to achieve per vertex lighting only.

You can see the result of tesselation in the "lighting" demo in the Dx9 SDK. More polygons means more pleasing interpolation and if done right should not cut performance too much.
Of course you can exclude additional tesselation from the silhouette calculation, since it doesn't add any geometric detail.

Be careful of saturation too. If your levels of lighting are too high, the result gets cut to the maximum lighting value (1.0), so most of the surface exposed to light gets maximum lighting and only a small number of your triangles show the transition between light and dark.
(in that case a higher tesselation wouldn't hurt either)

 

[KPixel]

I've fixed some problem with my sample.
Here is the new version : http://esotech.free.fr/pub/ExKalibur_v0.19.zip

more polygons wouldn't hurt especially
if you try to achieve per vertex lighting only.

The lightning is rather jagged because the 3DS file used contains simple objects.
Putting complex models make debugging harder...
I'm just testing my engine, not doing a real usage of it.
With more tesselated models, edges are less jagged... And with texture + game-like environnement, I think it'll be hard to notice.

I haven't fix the problem of distortion on the cuboids...
For the Teapot, it's not the fault of the engine; the Teapot isn't correctly closed (as I said...)

Move slightly to the right from original position, view stencil shadows and possible silhouette, and stop the light when the shadow from the sphere falls on the big pink cuboid. You shouldn't be able to see that shadow, but it's still visible.

It was because the "big pink cuboid" was not always CCW, so front faces were culled sometimes.
I correct this with "CULLMODE = CULL_NONE" (in a controlled environnement, I will set it to CCW).

When watching the "Possible silhouette" mode, the extruded edges goes in wrong direction. It looks like each vertex on the object is matched to wrong extruded vertex (the next vertex clockwise on the silhouette).

I've also found the problem here : I used the same ShadowVolumeVB to render "possible silhouette" (in LINELIST).
That's not correct because the vertices ordering don't allow this. That's why each vertex was associated with the next extruded vertex.
I've written another algo to render possible silhouette (less efficient ).
Note : To correctly see possible silhouette, you can press 'X' (models will only rendered with ambient light).



I am pretty happy now, but I have got some others (optimization) problems :

1- Generally, Dynamics VBs are shared with all geometries (as long as it’s the same FVF).
The algo to render n animated characters can be :
- ForEach Character
- - Lock & Fill the DynVB (with the animated vertices)
- - Render

But with the multi-pass algo of Stencil Shadow, it is very inefficient : We compute many times animated vertices.
I know "blending in Vertex Shader" can (partially) solve the problem, but I am a bit annoyed


2- I have a problem with determining the size of the Shadow Volume VB.
Actually, I take the size of the biggest VB and I multiply this size by 6 (a quad = 3 vert * 2 tris). That's because I take the worst case where ALL edges are extruded.
Actually, I can live with this method (because this ShadVolVB is shared by all geometries and its size is relatively small).
I should cache the ShadVolVB per geometry; as with DynVBs, it's a trade-off between performance and memory...


3- What's the simplest (but correct enough) way to handle the situation where the camera is IN the shadow ? I mean, to know that the camera is in the shadow.


Now, I'm going to build a game-like sample, to see how slow my engine is...