Bad aliasing issue.

[Area593]

Hi Everyone,

I'm having a really bad issue with edge aliasing, and I'm wondering if anyone knows of a solution. Here is an example:

As you can see the edges are being ripped to shreds. I'm running it on a 9600GT, and have tried multiple driver versions (except the ForceWare drivers) to no avail. It isn't an issue at all on other machines such as one of my laptops running an ATI Mobility Raedon HD 3650, and although I do get aliasing on that machine, it is "normal" aliasing which I expect with AA turned off.

Could it be an issue with running both the 9600GT and the inbuilt 6100 at the same time? I've triple checked all of the shader code, and engine code and it just keeps doing it, although as I said it's not an issue on my other machines. It only seems to do it with edges that are at a certain angle, so it's kind of got me stumped. I also haven't been able to find any info from my internet searches.

The engine is running Direct3D 9, and is pretty standard all the way through, nothing out of the ordinary as far as the engine and shaders.

Anyone know why this is happening? Thanks in advance for any help you can provide :smile:

 

[andypski]

Hmmm - I can't tell you exactly what's happening, but from a quick inspection it appears that the artifacts occur only in the top-left of the image - if you draw a line from the bottom-left to top-right of the image then everything to the right and below that line seems to look ok (as far as I can tell), and everything on the other side has strange artifacts.

Without more details it's hard to diagnose, although whatever is happening it appears to be related to some full-screen operation rather than anything happening during the rendering of the objects in the image(i.e. the errors are introduced in some image blit or post-processing operation) - there appears to be some jitter going on in the top-left of the image. It looks like some parameter is running into the limits of its precision and introducing the jitter, although I can't say why that should be happening.

Is there some sort of full-screen post-processing or compositing operation going on?

 

[Andrew Lauritzen]

I've seen similar things happening when doing a non-uniform stretch blit... As the above poster mentioned though, you may have just messed up your texture coordinates or sampling on some full-screen pass though (i.e. doing a non-uniform scale). If you are doing a full-screen pass, you should be using a single triangle that spans the entire screen though rather than a "quad" (two triangles) for performance reasons anyways.

 

[Area593]

You both seem to be right, there is an issue with the post processing. Thanks for the help.

I tried bypassing the post processing in the PS code before I started this thread and still had the issue. But I just tried completely bypassing all of the post processing altogether, and rendering the models straight to the output frame buffer, and it fixed it.

With the post processing I'm using a full screen quad, with the upper left corner at x= -1, y= 1 (u=0, v=0) and the lower right corner at x= 1, y= -1 (u=1, v=1), and am using this method for all stages of the post processing (all luminance and bloom stages). I also found that I had to use a different projection matrix in order to make that work (aspect ratio at 1), otherwise it stretched the output, whereas the projection matrix used when rending the models is based on the width and height of the frame buffer, as I would normally do it. Is this the correct way to do it?

I've also double checked the render target sizes for all post processing stages, but they are set up to scale relative to the size of the output frame buffer (the bloom stages are half the size of the frame buffer for performance) and even if I hard code the sizes it doesn't make a difference (even with the final bloom texture not being used in the final pass PS the issue is still there)

One other thing to note is that when rendering at 800x600, which is the windowed size I used when I started on the engine, the issues seem to go away, which makes me think it's not an issue with the geometry or texture coordinates of the full screen quad. I've also checked the code nearly twenty times, over and over again, and am yet to find the cause of the issue.

As far as the final pass, the texture that the models are rendered to is using point filtering (which I though was the logical option to prevent blurring), while the bloom texture is using anisotropic filtering to prevent artifacts when scaling it up to the final frame buffer size.

I don't want to seem like an idiot, and I know it pretty much has to be something simple, but I've never developed an engine with post processing before, so it's new territory to me.

Thanks for any help you can provide

 

[Davros]

"except the ForceWare drivers"

all drivers for the 9600gt are ForceWare drivers, so what exactly have you got installed ?

 

[Area593]

Hmmm - I can't tell you exactly what's happening, but from a quick inspection it appears that the artifacts occur only in the top-left of the image - if you draw a line from the bottom-left to top-right of the image then everything to the right and below that line seems to look ok (as far as I can tell), and everything on the other side has strange artifacts.

I've just realised it looks like you are right, it only seems to affect the upper left triangle that makes up the full screen quad, I'm still not sure why though.

 

[Area593]

"except the ForceWare drivers"

all drivers for the 9600gt are ForceWare drivers, so what exactly have you got installed ?

OK, sorry, my bad

I havn't done 3D stuff since nVidia had standard and detonator drivers, and I though they might be still doing something similar :smile:

 

[Area593]

It's fixed!!!

I still don't understand exactly what was happening, but seeing as the upper left triangle seemed to be the issue, I enlarged the lower right triangle to fill the screen, and modified the texture coordinates to suit, and it has resolved the issue, WOOHOO!

Thanks for all your help, and for putting up with my stupidity.

 

[Area593]

It's fixed!!!

I still don't understand exactly what was happening, but seeing as the upper left triangle seemed to be the issue, I enlarged the lower right triangle to fill the screen, and modified the texture coordinates to suit, and it has resolved the issue, WOOHOO!

Thanks for all your help, and for putting up with my stupidity.

I spoke too soon, unfortunately. There is still an issue. I don't know if it helps, but when the MagFilter is set to Linear the issue goes away, at the expense of the result becoming slightly blurry, and when it is set to point it comes back.

 

[Kaotik]

"except the ForceWare drivers"

all drivers for the 9600gt are ForceWare drivers, so what exactly have you got installed ?

Just to nitpick, haven't they been called just "GeForce drivers" for quite some time, ForceWare before that and Detonator before that?

 

[Andrew Lauritzen]

What kind of vertex shader are you using with these passes? Don't use a "projection matrix" or any other nonsense... just pass through your vertex locations directly. Even better, if you're using DX10 you can just compute the vertex positions in the vertex shader and forget the vertex buffer entirely.

Also if using DX10 use Load() together with SV_Position in the pixel shader to load the data from the back buffer. Otherwise you need to make sure you're texture coordinates are aligned with your texel positions (i.e. centered at half pixels). In DX9 this requires some fiddling with the polygon corners since the rasterization rules are a bit wonky. in DX10+ or OpenGL it "just works out".

Note that if you get all this right, you should get exactly the same results with point filtering as with linear (no blurring).

 

[Area593]

What kind of vertex shader are you using with these passes? Don't use a "projection matrix" or any other nonsense... just pass through your vertex locations directly. Even better, if you're using DX10 you can just compute the vertex positions in the vertex shader and forget the vertex buffer entirely.

Also if using DX10 use Load() together with SV_Position in the pixel shader to load the data from the back buffer. Otherwise you need to make sure you're texture coordinates are aligned with your texel positions (i.e. centered at half pixels). In DX9 this requires some fiddling with the polygon corners since the rasterization rules are a bit wonky. in DX10+ or OpenGL it "just works out".

Note that if you get all this right, you should get exactly the same results with point filtering as with linear (no blurring).

Unfortunately I'm using DX9 for this engine, as I use my main production machine to run both of my audio engineering and production related businesses, and the pro audio world is yet to fully support Windows 7 (don't get me started on Vista, lol). As a result, until I can afford to purchase a new machine specifically for games development I'm pretty much stuck with WinXP and DX9.

What you've said about aligning the texture coordinates to be centered at half pixels makes so much sense it hurts . When I finish up with the audio jobs in the studio today I'm definitely going to get that sorted. Thanks for the help! :smile:

I am looking forward to getting a games dev machine sorted simply so I can move on up to DX11 as there is a lot of features I'm looking forward to playing with.

 

[Area593]

OK, I got to the point where I couldn't concentrate on work, so I decided I had to try aligning the texture coordinates to the texel positions and it works like an absolute charm. I even tried it at multiple resolutions in both windowed and fullscreen (to avoid eating my words yet again), and it works awesomely!

Andrew, you are a legend, my brain can now work in peace. Thanks again!

 

[Rys]

If you are doing a full-screen pass, you should be using a single triangle that spans the entire screen though rather than a "quad" (two triangles) for performance reasons anyways.

Isn't the cost going to be really negligible for two really big triangles? When bringing up B3D benchmarks years ago, I measured perf with a quad and with a zoomed in tri and there was nothing in it, so quad it was for texcoord simplicity.

 

[Area593]

Isn't the cost going to be really negligible for two really big triangles? When bringing up B3D benchmarks years ago, I measured perf with a quad and with a zoomed in tri and there was nothing in it, so quad it was for texcoord simplicity.

That is kinda what I thought. I know it has to process one additional vertex, and three additional indices (for a triangle list), but in the end it works out to the same amount of pixels, which is where I thought the largest cost would be in any case. Though this is just me speculating, I've never tested it myself.

 

[Simon F]

The "enormous" triangle approach might even be worse if the guard-band region was not large enough to fully contain that triangle resulting in clipping. <shrug>


BTW, regarding the topic. Can one ever have a "good aliasing issue"

 

[Andrew Lauritzen]

Happy to help Note that this (and a lot of other things!) got cleaned up in DX10, so hopefully you can get on board soon!

Isn't the cost going to be really negligible for two really big triangles? When bringing up B3D benchmarks years ago, I measured perf with a quad and with a zoomed in tri and there was nothing in it, so quad it was for texcoord simplicity.

The cost is not in the vertex transform/interpolation but the redundantly shaded/unused SIMD/quad lanes at the triangle boundary (diagonal across the screen). As such, it fully depends on the SIMD size of the machine and the cost of the shader at those pixels.

The "enormous" triangle approach might even be worse if the guard-band region was not large enough to fully contain that triangle resulting in clipping. <shrug>

I can't imagine the clipping would ever be slower than the extra shaded pixels for a single triangle... even *if* the guard-band didn't cover just 2x outside the screen.

I've seen a single triangle make it go a bit faster, and I've never seen it make it go slower Not really a big deal, but it's easier to deal with a single primitive too.

 

[Simon F]

I can't imagine the clipping would ever be slower than the extra shaded pixels for a single triangle... even *if* the guard-band didn't cover just 2x outside the screen.

I did say "might"

 

[Raqia]

BTW, regarding the topic. Can one ever have a "good aliasing issue"

What if you're writing a screen saver and want a cheap way to do Moire patterns?

 

[Hazuki Ryu]

I have a similar problem in Divinity II.

It makes it look pretty weird, any suggestions for a fix?

http://img242.imageshack.us/img242/4448/divinity220100305204644.jpg

HD5850 drivers 10.3 beta, the same happened on the game demo and back then I had a 9800GT.