Lots of interesting stuff...
What I find interesting is that in its original article ET claimed to see no rendering abnormalities under the Catalysts of the type he exposed in the Detonators, but apparently the idea that ATi *isn't doing the same thing nVidia is doing* in its drivers now completely escapes him. It's surprising he doesn't see that the fact that rendering problems were evident with the Detonators but not with the Catalysts indicates two entirely different approaches to the benchmark.
All I can say is that it's also interesting that ET's reported findings directly contradict FutureMark's own published findings, which state the aggregate differences in score as being around ~1.8% for ATi. This corresponds with my own obervation of a 5860 score running the 320 version and a 5774 score running 330 with Cat 3.4's.
Under the identical conditions to the ATi products in the FutureMark test the 5900U drops 24%+ aggregate when the Detonators can no longer recognize that they are running something other than a 3D game when 3D Mark is run. I thought it was somewhat amusing that although he quoted a great deal of the FM report he decided not to quote their results and their comments about those results. And I found it interesting to say the least that he restricted his testing to two resolutions (and did not use the 3d Mark 03 default resolution and FSAA levels, which a majority of 3D Mark users use I would imagine.) I am also leery of directly comparing FSAA benchmarks of *anything* unless it is thoroughly established that the IQ delivered is pretty much the same between products--he says nothing about that--that sort of thing should be foundational to any benchmark comparisons which rely on FSAA mode comparisons since without establishing at least a rough equality of IQ any FSAA results so obtained are suspect until proven valid. And yet, here is how ET chose to test:
We tested at 1,280x960x32 and 1,600x1,200x32, and with both 4X AA and 8X AF enabled at both resolutions
I find it most interesting that in ET's first report exposing the cheats ET did not feel that high-resolution or FSAA mode comparisons were important. At all.
We know for a fact that nVidia and ATi use different methods for their respective FSAA modes, so it goes without saying their IQ is bound to be different. IE, nv35 FSAA != R9800P FSAA and so "IQ sorting" is essential for any accurate comparison. I also wonder why some folks are shying away from 6xFSAA/16xAF 8xFSAA/8xAF comparisons--which in my view are just as valid as any 4xFSAA comparisons, for the simple reason that these settings represent the best objectively determined IQ settings for each product as established by their respective manufacturers.
I use 6xFSAA/16xAF routinely in a number of 3D games I own because in those games the performance is fine and the resulting IQ is terrific. Right now it looks like Brent's article at [H] is the only one which has nailed that down--with the somewhat surprising results that the R9800P is on average 2x as fast as a 5900U at those settings, yet delivers demonstrably better IQ at the same time! This leads me to deep suspicions as to what nVidia's doing at lower FSAA modes, and illustrates why you can't just compare 4x FSAA modes on the two products without seriously looking at IQ to establish that it is indeed comparable.
Basically, what do FSAA mode fps performance comparisons mean between products without a strict IQ examination? Just about nothing, IMO.
Tell the truth, this article from ET seems as much an apology for his previous article as it does anything else--for some reason he feels it necessary to go to lengths to prove the two products a "wash" without ever actually proving that they are indeed a wash in reality. The politics of the situation seem to have affected his ability to do what needs to be done and his desire to do it, which I think is unfortunate.
If this situation should teach anyone any lesson at all it is that bar charts can lie, and that there is much which is hidden and goes on behind the scenes when a bar-chart graph is constructed. Without a competent examination of the issues which underlie bar charts--whether for a 3D game or a benchmark--such as IQ, for instance, as just one of many things, the bar charts are meaningless.
As it is, because there are relative IQ questions that have yet to be sufficiently answered with respect to these issues, I think the FutureMark numbers should stand as the definitive word on the subject:
(in bold--Lifted verbatim from the .pdf)
In our testing, all identified detection mechanisms stopped working when we altered the benchmark code just trivially and without changing any of the actual benchmark workload. With this altered benchmark, NVIDIA’s certain products had a performance drop of as much as 24.1% while competition’s products performance drop stayed within the margin of error of 3%. [Emphasis mine]
....
A test system with GeForceFX 5900 Ultra and the 44.03 drivers gets 5806 3DMarks with 3DMark03 build 320. The new build 330 of 3DMark03 in which 44.03 drivers cannot identify 3DMark03 or the tests in that build gets 4679 3DMarks – a 24.1% drop.
It's pretty obvious to me that FutureMark wasn't involving itself in gray areas of FSAA-mode IQ comparisons (only gray because they have yet to be solidly pinned down) and based its results on the default mode of the benchmark (which by far is the most used among the benchmark's user base.)
I'm not saying that high-res, FSAA comparisons don't matter because of course they matter very much--I'm saying that until it can be determined what FSAA mode is actually comparable with what on an IQ basis (which is what FSAA is all about), performance comparisons between FSAA modes which fail to do this are bankrupt.
