Anti Aliasing and alpha textures, maybe this might work?

[OpenGL guy]

It doesn't look good.

As expected, as it would turn alpha-blending into dithering (too few transparency levels), and also, likely select the same sample points for different polygons for the same pixel, causing transparent layers beneath the top layer to disappear.

It should work fine with alpha-testing, though.

I don't think so. Imagine a texture where alpha values are either 0 or 1. Your coverage mask would then always be either 0 or 1, hence, no AA.

Umm, texture filtering? You would usually apply bi/trilinear interpolation to the texture map, so unless texels are perfectly pixel aligned, you would get alpha values between 0 and 1 even if the texture map itself only contains 0s and 1s.

You said alpha test. If your alpha ref value is 1.0, it doesn't matter that filtering gives alpha values less than 1: You won't see those texels.

That was why I said I don't think this works.

Edit: Grammar.

 

[arjan de lumens]

Yes, I can see that the alpha-to-coverage-bitmap trick won't work too will with an alpha test reference value of 1.0.

But: if you set the reference value to 1.0, the edges introduced by the alpha test will be severely pixelated, similar to what happens if you use point sampling instead of bilinear interpolation. Also, if you do mipmapping with that setting, any holes in the alpha-tested polygon will either shrink or grow for each mipmap level, looking rather weird. Is this really at all common to do in real life? I would have expected reference values around 0.5 to be more common in practice?

 

[OpenGL guy]

Yes, I can see that the alpha-to-coverage-bitmap trick won't work too will with an alpha test reference value of 1.0.

But: if you set the reference value to 1.0, the edges introduced by the alpha test will be severely pixelated, similar to what happens if you use point sampling instead of bilinear interpolation. Also, if you do mipmapping with that setting, any holes in the alpha-tested polygon will either shrink or grow for each mipmap level, looking rather weird. Is this really at all common to do in real life? I would have expected reference values around 0.5 to be more common in practice?

No idea, but I don't think it works for 0.5 either because you'll get a harsh (i.e. non-AAed) boundary at the edge in any case because you'll be going from 0.5 to 0 (i.e. transparent) all at once.

 

[arjan de lumens]

Oh .... OK, now I finally see the problem: after the alpha-to-coverage-bitmap conversion, all the samples that result from the coverage-bitmap application for the pixel have the same alpha, so they will either all pass or all fail the alpha test, resulting in no AA for edges produced by alpha testing

 

[Xmas]

Alpha-to-coverage is a quick-n-dirty replacement for alpha blending, you shouldn't use alpha test there except for saving bandwidth on fully transparent parts of the texture.