Rounding behavior of pixel shader output

51mon

Newcomer
Hi
I have searched the internet for a while but can't find the answer. I want to know what the rounding behavior of the output to RGBA (8-bit per channel) is? The platforms I'm interested in is XBOX360, PS3 and Windows OpenGL. I seam to remember that it was floor function in DirectX but am not sure.

Thanks
 
I believe the answer is it depends. It depends on what operation you're doing and what mode you've set it to.
 
I believe the answer is it depends. It depends on what operation you're doing and what mode you've set it to.

Are you referring to blend operations? Sorry, I didn't really understood your answer.

For example:
Code:
float4 colResult;

// Fill colResult...

gl_FragColor = colResult;

If the render target got 8 bit precision per channel, how is the rounding of colResult performed? Can I control the behavior?
 
I am pretty sure that you will get the nearest in 8-bit representable number to colResult with GL, but I am too lazy to look through the specs right now (maybe you could).
 
I am pretty sure that you will get the nearest in 8-bit representable number to colResult with GL, but I am too lazy to look through the specs right now (maybe you could).

Sure, definitely. Do you know where I can find those specs? Sorry, I'm an old DX user :oops:
 
Both DirectX and OpenGL use round to nearest. It's the only correct mode to minimize precision loss.

In C code:
Code:
f = max(min(f, 1.0f), 0.0f);
i8 = (unsigned char)(f * 255.0f + 0.5f);   // C cast uses round towards 0
 
Back
Top