4x AA performance comparisons invalid?

[Fuz]

I recall discussions on these forums regarding review methods, and the correct way of comparing two cards in terms of image quality and performance.

Most agreed that the image quality should be as similar as possible when comparing performance between two cards. For instance, if 32xAF on card A produces the same image quality as B's 2xAF, then A's 32x should be compared to B's 2xAF.

Well, the general consensus seems to be that ATI's AA seems to produce better quality than the FX's.

Take a look at the following images for instance.

NFS R300 2xAA
NFS NV30 4xAA

UT2003 R300 2xAA
UT2003 NV30 4xAA

Now the images look pretty similar don't they? So should we be comparing ATI's 4x to Nv's 4? Or should it be ATI's 2x to Nv's 4x?

Ok, now ofcourse many of you will say that Nv's 4x is better than ATI's 2x in those shots I linked to. Well, would you say Nv's 4x is as superior to ATI's 2x as ATI's 4x is superior to Nv's 4x?

Imho, comparing performance of NV30's 4xAA to R300 4xAA is not really a fair comparison. So whats your opinion?

 

[Doomtrooper]

Yes Anand tried to do that with Anisotropic filtering, by getting a baseline IQ comparison..but not done on FSAA.
I agree, we need to establish a baseline IQ..why should one card 'work harder' at a higher setting when the IQ is much higher with the sliders matched. Or the same 'why should a card not have to do as much work' with sliders matched if the IQ is lower'

Baseline IQ again meaning approximations, not zoomed in 100% to compare settings..common sense.

 

[Brent]

Yes Anand tried to do that with Anisotropic filtering, by getting a baseline IQ comparison..but not done on FSAA.
I agree, we need to establish a baseline IQ..why should one card 'work harder' at a higher setting when the IQ is much higher with the sliders matched. Or the same 'why should a card not have to do as much work' with sliders matched if the IQ is lower'

Baseline IQ again meaning approximations, not zoomed in 100% to compare settings..common sense.

sometimes, but not always, it is subjective until you zoom in to see what it is doing...

 

[Arun]

I'd suggest comparing to a lower quality ATI Anisotropic mode ( such as 4x Performance ) to compensate for the slight AA IQ difference.
4x AA from nVidia sure looks worse than ATI's 4x AA, but it still looks slightly better than ATI's 2x IMO.


Uttar

 

[Fuz]

4x AA from nVidia sure looks worse than ATI's 4x AA, but it still looks slightly better than ATI's 2x IMO.

Well, that is why I asked;

Ok, now ofcourse many of you will say that Nv's 4x is better than ATI's 2x in those shots I linked to. Well, would you say Nv's 4x is as superior to ATI's 2x as ATI's 4x is superior to Nv's 4x?

 

[Doomtrooper]

Its starts getting subjective for the reviewer..thats why including the screenshots of all modes allows the viewers to make their own decision.
They will get disected anyways on the forums, people pointing out what is important to them.
Yet common sense needs to be used when comparing and using a baseline IQ, the reviewer matches it as close as he can between the two cards (shows his screenshots and settings to prove it) and start benchmarking.

Baseline is now done
Now benchmark all the modes
This type of review covers two things:

1)Comparison with IQ is centered on
2)All other modes are covered also.

4X on one slider doesn't meet 4X on another...as shown in screenshots.

 

[demalion]

Too much subjective evaluation if you go to far. I think Anand did it the right way (barring the GF FX 2x AA shots being incorrect still AFAIK) for the first half, where the image quality is presented and discussed and compared. The only thing lacking there, I think, is the "Application mode" for the GF FX.

Then, as Anand did, the reviewer's opinion of what image quality they consider comparable is outlined as a result of these shots, along with reasoning (I think he must read these forums, ), and then comparison going forward is based on that, with all the data presented, not just what the reviewer considers applicable. Again, I think the "Application mode" is missing.

What Anand's review lacks in the second part, IMO, is some sort of aid in placing the "apples" to "apples" comparisons close together per the user's own choice (like some sort of custom graphing application). Strikes me as something that could be done if someone wanted to spend the time doing so. The simplest way I can think of doing it is to have the user pick the images they think equivalent, and color those bars in the graphs with highlighted and brightened colors throughout the rest of the review (with a palettized image format, this would be easy, I think, if a scripting language had the capability to change palette color values).

The best solution would be to order the fps graphs by image quality as determined by the user.

 

[Nite_Hawk]

Really, I think it's up to the reviewer how and what they want to benchmark, but the only reviewers I have any respect for are the ones that clearly state why they are performing the benchmark in a specific manner, and detail exactly how they've achieved the results they've gotten. If the reviewer thinks comparing the GF FX's 2X aa to the Radeon 9700's 6x yields interesting data and explains why, I don't mind at all.

You know, not to rant, but reviews still have a lot of room for improvement. (though places like beyond3d and now hardocp seem to be doing pretty well.) We need standard deviations for framerate scores, image analysis comparisons (not just subjective, but quantitative), and of course subjective opinions as well. This is the kind of information I'd actually be willing to *pay* for. Some of the crappier reviews out there aren't even worth my time to read let alone money to pay the writer.

Nite_Hawk

 

[Bambers]

I'd say nv30s 2x is pretty similar to the r300s 2x. Since they're the same pattern (except nv30 2x is \ way and atis 2x is / way) the only difference is gamma correction.

r300 4x to nv2x/30 4x is not a fair comparision imo.

 

[THe_KELRaTH]

Reading ATI's blurb on their AA method it says that it uses programmable sample patterns. On that basis would that not mean that in every angle at say x2aa it will always be a true x2 whereas on a fixed grid there will always be angles that will get less than x2aa and with x4aa fixed grid there will be angles where it's only actually performing x2aa

 

[Dave H]

Reading ATI's blurb on their AA method it says that it uses programmable sample patterns.

R300 does not currently use programmable sample patterns. The capability is designed into the hardware, but is not enabled in current drivers.

 

[KimB]

The only problem with attempting to set a baseline FSAA image quality level for performance comparisons between the Radeon 9700 and GeForce FX is that the only time the two are very close to equal is when both are running with 2x FSAA.

 

[Windfire]

The only problem with attempting to set a baseline FSAA image quality level for performance comparisons between the Radeon 9700 and GeForce FX is that the only time the two are very close to equal is when both are running with 2x FSAA.

Are we seeing the same pictures here? The screen shots supplied above clearly show that ATI's 2x is easily equal to Nvidia's 4x, let alone Nvidia's 2x.

 

[Windfire]

Now the images look pretty similar don't they? So should we be comparing ATI's 4x to Nv's 4? Or should it be ATI's 2x to Nv's 4x?

Someone should setup several blind tests with Nvidia's 2x and 4x and ATI's 2x and 4x. Let users pick the order from "best to worst or tie" or something.

Looking at those pictures I had to double-check the pictures to confirm that I actually thought ATI's 2x looks better than the 4x! I would almost question the legitimacy of those pictures!

 

[OpenGL guy]

Someone should setup several blind tests with Nvidia's 2x and 4x and ATI's 2x and 4x. Let users pick the order from "best to worst or tie" or something.

I doubt it would work. For one, people could use magnifying glasses to see which had more samples. Thus, results wouldn't necessarily be based on visual quality, but on how many samples were used.

 

[Nagorak]

I'd suggest comparing to a lower quality ATI Anisotropic mode ( such as 4x Performance ) to compensate for the slight AA IQ difference.
4x AA from nVidia sure looks worse than ATI's 4x AA, but it still looks slightly better than ATI's 2x IMO.


Uttar

Any difference between the 2X and 4X shots is so slight as to be nonexistant. If you had only 10 seconds to look at both shots, you wouldn't be able to tell them apart.

My question about comparing quality is: to what end? Will you compare 2X AA on R9700 to 4X on the FX in terms of performance, or what?

 

[MrNiceGuy]

ok...

Since nv 2x & ati 2x are pretty much the same, then by that argument, one should compare nv's 2x to nv's 4x??

At low #s of samples, sample placement is very important to edge quality, which is why 2x ~= ordered 4x in general.

For roughly equal quality, you would have to compare nv's 4xS to ati's r300 4x.

 

[noko]

My opinion is to run the benchmarks, show the IQ differences and make an assessment. What I've have seen so far is ATI's 4x AA for the most part is superior to any other mode on the GF FX. ATI's 2x is superior to Nvidia's 2x and looks close to Nvidia's 4x. What shocks me more is that my GF3 Ti200 looks better at 2x and 4x then the GF FX.

 

[THe_KELRaTH]

Any idea as to why at 80 to 100 and 135 to 155 degrees Nvidias AA is almost non existant even at 8xS level?

 

[Doomtrooper]

I thought the way Brent did it was idealy the best way, allow the readers to make their own opinion by showing screenshots...when it comes to performance well its not ideally correct to have one card looking like crap and the other great and comparing performance numbers..

With screenshots hopefully the readers would be wise enough to realize that its not a apples to apples comparison.