[PeterT]

I'm currently implementing an algorithm that would benefit from using the built-in FP32 linear filtering on g80. However, testing has revealed that this filtering has deviations of 5%-10% from my software filter implementation.

Does anyone know how and with which precision FP32 bilinear filtering is implemented on G80? Note that I'm not doing any minification or magnification, just offsetting by a sub-pixel amount.

(If you've read this post before the edit please ignore what you read then )

[nAo]

Can we see your software implementation?

[armchair_architect]

I don't think any mainstream gfx cards use full fp32 precision for the filter weights in the bilinear filter. I think its more like 8-12 bits of precision.

You can test this easily enough: create a 2x1 texture with black in one pixel and white in the other. Then map it onto a big wide quad, with texcoords 0.0 on the left and 1.0 on the right, and bilinear filtering. The number of different shades of gray you get tells you the filter precision (256 shades => 8 bits).

[Andrew Lauritzen]

I believe the alpha value in the interpolation is truncated somewhat; I had trouble with the same thing when playing with some real-time Bezier surface raytracing.

So while the values drawn from the texture seem to be in full fp32, and the interpolation evaluated as such, there seems to be a limit in the precision of the weight parameter... this may be what you're experiencing.

[Edit] Whoops I see that's what armchair_architect said too. Now I'm really convinced
By "mainstream graphics cards" are you implying that the "professional" cards (Quadro, FireGL) handle this differently?