Newbie Vertex Shaders and Fog Question

[Arshad]

Hi, I'm just starting off with vertex shaders and I'm a bit confused about fog. Specifically, I use standard GL_FOG, and everything works as expected. I then switch to ARB_vertex_program with a simple shader that just does the transformation matrix multiply and passes through texcoords and vertex color straight through. It works as expected, but the fog color is no longer applied. I'm a little confused because I thought that fog was still a fixed function stage after the fragment shading pipeline.

My understanding is that I can use per-vertex fog using EXT_fog_coord or something, but that's not the route I want to go. I just want to use the standard GL fog. Do I need to do something specific in the vertex shader in order to allow fog to work properly? BTW, this is on a Radeon8500 using 7.79 drivers. Any ideas?

 

[Hyp-X]

Fog is both a vertex and a fragment operation.
The amount of fog for the vertices should be calculated during vertex processing, in your case in you vertex shader program.

You should pass the computed fog alpha in an output register. (oFog in D3D, result.fogcoord in that extension if I read it right).

 

[Arshad]

Hyp-X, are you talking theoretically or in practice? I was under the impression that vertex.fog and result.fog were for vertex based fog calculations only. ie if I was using the EXT_fog_coord extension. In fact, when I try and compile using those without having EXT_fog_coord, the shader errors out.

I'm a little confused by the wording in the ARB_vertex_program definition:

"If EXT_fog_coord is not supported, the line "fog" should be removed from the <vtxAttribItems> grammar rule, and "vertex.fog" should be removed from Table..."

Next paragraph it says:

"If EXT_fog_coord is not supported, the fog coordinate output (result.fog) still operates as specified."

So what the heck is the input supposed to be? ie I can output result.fog, but what is the input if it's not vertex.fog?

I know there are a lot of demo/game coders here, so how do you guys do fog in conjunction with vertex shaders? Or do I need to manually simulate it by determining the distance of the vertex from the camera and then lerp'ing between 1,1,1 and the fog color or something?

 

[Hyp-X]

Hyp-X, are you talking theoretically or in practice?

Both.
I can talk about D3D in practice and about OGL theoretically (since I don't know it close enough).

In D3D you need to reproduce per vertex fog calculation in the vertex shader.
I don't think it works differently in OGL (as the hw is the same), but I can't tell you the details.

Or do I need to manually simulate it by determining the distance of the vertex from the camera and then lerp'ing between 1,1,1 and the fog color or something?

I think you need to simulate it, altough I don't get the calculation you describe.
Fog color has nothing to do with vertex processing, it's a constant applied during fragment processing.
The only thing changing per-vertex is the fog-blend value.
You only have to calculate that blend value.

The fog that is closest to reality is exp(-density * dist), where density is a constant, and dist is the distance from the view-point. There are other types of fog (exp2, linear, Z-based, W-based), so if you use the fixed T&L route too, you have to look up the functions used so you can perform the matching calculations.

Examples (D3D syntax):

; dist based exponential fog
; r0 is the vertex coordinate in camera space
; c0.x is the fog density constant

  dp3 r1.x, r0, r0
  rsq r1.y, r1.x
  mul r1.x, r1.x, r1.y
  mul r1.x, r1.x, -c0.x
  expp r1, r1.x
  mul oFog.x, r1.x, r1.z

; z-based linear fog
; r0 is the vertex coordinate in camera space
; c0 is {0, 1, fog_end, 1/(fog_end-fog_start)}

  add r1.x, -r0.z, c0.z
  mul r1.x, r1.x, c0.w
  max r1.x, r1.x, c0.x
  min oFog.x, r1.x, c0.y

 

[Arshad]

Fog color has nothing to do with vertex processing, it's a constant applied during fragment processing.
The only thing changing per-vertex is the fog-blend value.
You only have to calculate that blend value.

Ah ok, that explains it! That was the one essential concept I wasn't grasping Thanks for the sample code as well. Now my problem is that GL complains when I try to compile a program using result.fog. Maybe I need to use the EXT_fog_coord extension or something, I'll give that a shot. Thanks again!

 

[Arshad]

Duh. I should stop referring to my old draft of ARB_vertex_program and just look at the fully ratified spec. The problem I was encountering was trying to use result.fog (from older spec) instead of result.fogcoord, which is the correct way to do it. Now it works fine, with no complaints. Thanks again for your help Hyp-X!