@Iron: I'm using Quake's original light positions. Actually it's funny: ID created a pretty realistic lighting model but didn't use it that way. They kind of used lights to "decorate" area's, hence 200-300 lights per map. They could get away with that because they prerendered that data to lightmaps. So, in the end, an area in Quake may contain way beyond 24 lights. I just limit the number visible shadow casting lights to 24 per rendered frame (which is mostly ok). Besides that, each surface is limited to cast light from the 4 most nearby lights. So custom maps shouldn't break the engine, but may contain errors in terms of lighting. In contrast, a map that is designed with a limited, more realistic, number of lights will look better and is less likely to contain errors.
@Fafalada: Thanks! Since the stencils are exported to a texture I should be able to use hw filtering (i.e. implementing an actual 4x4 filter in tev would take 16 texture samples and require a complete 16 stage pass). Though the problem with hw filtering would be that the colour values must differentiate enough; there is no integer value between 0 (no shadow) and 1 (shadow) which means a shadow value must be stored as a value bigger than 1, so the hw can actually interpolate between those. From there it's just multiply the filtered value with the light intensity. Yes, I'm currently using RGB8 but I export to RGB565 to reduce texture size (which means I have either 5 or 6 bits available). There are some other formats available that may suit better but I still have to test with those.
@Deadly: pretty much what Tron says. Dedicated bump maps will enable the same results as I showed with the cube demo. But I'd like to add that if a surface is lit uniformly from each side, you don't see the bumps (just like in a real world surface). Speculatity makes bumps more evident too, which you see in the first image you referred to.
@Link: That's what I'm planning to do. But, with the shadowing going on it runs only 12 fps. With some optimizations it runs 16 fps, but with a lot of errors in shadowing. I estimate that this can run about 25 fps at least, but it requires optimizing the shadow volumes first (a shadow BSP would be a good point to start). When I have managed to do that I'll release that version. The followups will contain shadowing and normal mapping on entities and support for texture and light overriding. Hopefully I can provide a download end next month or so (I possibly can't work fulltime on it the next 7 weeks since I ran into an assignment I just had to take on).
Sadly enough, no new screens yet. It's hard to make good progress with this stuff!
