New cheats exposed at DigitLife
- Thread starter BoardBonobo
- Start date
Looks like hype to me. ATI did remove the 3D Mark 2003 GT4 stuff. Tests have shown this. Since that was the basis for the whole article at Digit-Life, this sounds like garbage.BoardBonobo said:
One thing that occurred to me is that the Radeon 9800 has 24-bit floats. If the refrast uses 32-bit floats and the GeForce FX uses 32-bit floats with detection disabled, that could easily account for the difference in results.
-FUDie
While not actually reading the comments I note that they talk about comparing with the ref rast. While the ref rast is indeed "the reference", making direct per pixel comparisons with it can be quite tricky and even potentially incorrect. One issue :
RefRast will do all maths at 32bits IEEE... R9800 is 24bits... so there are bound to be differences. There are a lot of other small tidbits that can go wrong or cause differences... there are lots of other accuracy issues that can show, texture interpolators, iterators, filtering accuracy...
So its not a trivial step to compare with ref rast and claim cheating if there are some difference present. Judging if ATI and or NVIDIA are within the margin of error can really only be answered by comparing the hardware spec and the ref rast spec and only ATI can do that.
Just my 2c based on not really reading the comments...
K-
RefRast will do all maths at 32bits IEEE... R9800 is 24bits... so there are bound to be differences. There are a lot of other small tidbits that can go wrong or cause differences... there are lots of other accuracy issues that can show, texture interpolators, iterators, filtering accuracy...
So its not a trivial step to compare with ref rast and claim cheating if there are some difference present. Judging if ATI and or NVIDIA are within the margin of error can really only be answered by comparing the hardware spec and the ref rast spec and only ATI can do that.
Just my 2c based on not really reading the comments...
K-
That's one big problem with the diff images they are using. Since they are boosting the contrast, how can you tell if the contrast is being boosted equally on each shot? For example, on some images there are such large differences (like the 44.03 PS 2.0 test result) that the contrast doesn't need to be boosted to see the difference.Kristof said:
ATI has released numerous DX9 WHQL'd drivers for the R300 based parts, so I guess they are in spec.
-FUDie
I love this excerpt from the article:
Anybody with a slight technical understanding (which as I recall they would profess to have) can answer this one for them. Of course it is possible, in fact it is overwhelmingly likely.
The fallacy of attempting exact bitwise comparisons between hardware and reference images has been shown in the past. There are minor mathematical differences between all hardware and the reference, even in the case where all the precisions are the same - it's pretty much unavoidable with implementations that use floating point numbers. One clear example is that the reference rasterizer implementation of dot product operations almost certainly differs significantly from all hardware implementations. Since dot products are one of the most common operations you can imagine in 3D graphics you are going to get slight differences in a lot of calculations.
These differences are then magnified when you consider going through a long pipeline of multiple stages with slight differences in order of operations, slight variations in texture filtering etc.
Anybody with a slight technical understanding (which as I recall they would profess to have) can answer this one for them. Of course it is possible, in fact it is overwhelmingly likely.
The fallacy of attempting exact bitwise comparisons between hardware and reference images has been shown in the past. There are minor mathematical differences between all hardware and the reference, even in the case where all the precisions are the same - it's pretty much unavoidable with implementations that use floating point numbers. One clear example is that the reference rasterizer implementation of dot product operations almost certainly differs significantly from all hardware implementations. Since dot products are one of the most common operations you can imagine in 3D graphics you are going to get slight differences in a lot of calculations.
These differences are then magnified when you consider going through a long pipeline of multiple stages with slight differences in order of operations, slight variations in texture filtering etc.
That digit-life article seems very misguided. The Iq differences between the 44.03s and 44.65s are not important, its whether they are using application specific optimizations which Futuremark doesnt allow and ATI has made a point of disabling.
Actaully they seem to basically imply that they are. Also what ATI cheats are they talking about as said above theres always some minor differences to the reference rasterizor when doing complicated graphics.
Actaully they seem to basically imply that they are. Also what ATI cheats are they talking about as said above theres always some minor differences to the reference rasterizor when doing complicated graphics.
A pretty retarded article. As already noted, assuming that different vendors' output should be pixel-identical to the reference rasteriser is just nonsense. The reference rasteriser is good for checking whether there are relevant quality differences between different implementations, but doing image subtractions assuming a requirement of identical output is very misleading. Image subtractions are valid within different conditions on the same renderer, not between renderers.
Also, the language throughout the article is very crude (and I don't mean the grammar) and not at all proper for an article to be taken seriously. Statements like "Futuremark is lying" are dead wrong and actually pretty childish, FM has explicitly stated that the newest patch fixes the problems they are aware of, they have not proclaimed the version to be free of all possible cheats.
Also, the language throughout the article is very crude (and I don't mean the grammar) and not at all proper for an article to be taken seriously. Statements like "Futuremark is lying" are dead wrong and actually pretty childish, FM has explicitly stated that the newest patch fixes the problems they are aware of, they have not proclaimed the version to be free of all possible cheats.
Kristof said:
Wouldn't that also go the other way as well? For instance, they do refrast image checks on the CPU test, which uses PS1.4 - wouldn't the refreast be calculating PS1.x shaders in integer precision, whilst R3x0 would still be FP24?
I think everyone in the thread has summed up the issue adequately and would only add that if the difference between "cheat" and "optimization" isn't understood then an article of this type becomes an impossibility. I think the article does a far better job of refuting the viewpoint of its authors than it does anything else. It's sad, but I think this sort of techno-gobbledegook babble is perpetrated mainly to whitewash the site's earlier, equally indefensible, comments relative to the products of both companies on a comparative basis.
Heh...the encyclopedic ramblings of this site frequently reminds me of the old adage: "When all else fails, dazzle 'em with BS."
Heh...the encyclopedic ramblings of this site frequently reminds me of the old adage: "When all else fails, dazzle 'em with BS."
chavvdarrr
Veteran
Unbelievable IMHO
I just can't get it - all of you bashing the article because they dare to say that Ati used cheats too? Because they dare to say that R3xx image has differences with reference rasterizer(RR)? Posting images which show the difference b/w NV3x and RR is ok, whole threads were made.... but saying a word against the "strange", to say the least, position of FM is bad?
Sure, its not trivial step to compare with RefRast and claim cheating, and this IMHO is the problem with this article,- but maybe someone can give better "recipe" ? Better view on current problems with "testing videocards" - "how, why with what tools"?
[offtopic]While the article is far from perfect, IMO it's still the best one for now - showing how many questions are unanswered and giving idea how many things are unclear in this area.[/offtopic]
I just can't get it - all of you bashing the article because they dare to say that Ati used cheats too? Because they dare to say that R3xx image has differences with reference rasterizer(RR)? Posting images which show the difference b/w NV3x and RR is ok, whole threads were made.... but saying a word against the "strange", to say the least, position of FM is bad?
Sure, its not trivial step to compare with RefRast and claim cheating, and this IMHO is the problem with this article,- but maybe someone can give better "recipe" ? Better view on current problems with "testing videocards" - "how, why with what tools"?
[offtopic]While the article is far from perfect, IMO it's still the best one for now - showing how many questions are unanswered and giving idea how many things are unclear in this area.[/offtopic]
didn't this show that ati wasn't cheating in the 3.5s except for 2001 ? http://www.elitebastards.com/page.php?pageid=1551&head=1&comments=1
chavvdarrr
Veteran
more
I recommend reading (or at least watching) this article http://www.ixbt.com/video2/gffx-16.shtml (should be translated in 2-3 days)
It compares 5900 vs 9800Pro in :
Return to Castle Wolfenstein (MultiPlayer) — OpenGL, ixbt0703-custom demo
5900 look worse in this custom demo ...
Serious Sam: The Second Encounter v.1.05 (Croteam/GodGames) — OpenGL, ixbt0703-custom demo
9800 wins
Quake3 Arena v.1.17 ixbt0703-custom demo
5900 wins only without AA/AF or performance AF
Unreal Tournament 2003 v.2225 (Digital Extreme/Epic Games) — Direct3D, Vertex Shaders, Hardware T&L, Dot3, cube texturing, default quality
Definitely anti-detect stops some "fair" optimisations (difference in custom timedemo), definitely the often used demo shows suspicious results...
Code Creatures Benchmark Pro (CodeCult)
default vs Anti-Detect [%]
5900 -30%
9800P 0%
5900 vs 9800 -30% (with AD) translation mine.:
no "driver bugs" detected, 5900 loses in every resolution/setting
RightMark 3D v.0.4 (one of "game" tests)
no "driver bugs" 5900 is ~ 20% slower
I recommend reading (or at least watching) this article http://www.ixbt.com/video2/gffx-16.shtml (should be translated in 2-3 days)
It compares 5900 vs 9800Pro in :
Return to Castle Wolfenstein (MultiPlayer) — OpenGL, ixbt0703-custom demo
5900 look worse in this custom demo ...
Serious Sam: The Second Encounter v.1.05 (Croteam/GodGames) — OpenGL, ixbt0703-custom demo
9800 wins
Quake3 Arena v.1.17 ixbt0703-custom demo
5900 wins only without AA/AF or performance AF
Unreal Tournament 2003 v.2225 (Digital Extreme/Epic Games) — Direct3D, Vertex Shaders, Hardware T&L, Dot3, cube texturing, default quality
Code:
def vs AD
(1024x768 /1280x1024 /1600x1200)
5900 -14% -28% -32%
9800P 0.3% -0.2% 0%
def vs AD (antalus flyby)
5900 -23% -36% -38%Code Creatures Benchmark Pro (CodeCult)
default vs Anti-Detect [%]
5900 -30%
9800P 0%
5900 vs 9800 -30% (with AD) translation mine.:
Unreal II: The Awakening (Legend Ent./Epic Games) — Direct3D, Vertex Shaders, Hardware T&L, Dot3, cube texturing, default quality
no "driver bugs" detected, 5900 loses in every resolution/setting
RightMark 3D v.0.4 (one of "game" tests)
no "driver bugs" 5900 is ~ 20% slower
Re: Unbelievable IMHO
Am I an ATI fan boy? Not at all, I've used 3dfx and NVidia until today. My next card might very well be an ATI card, though.
The article claims:chavvdarrr said:
Yet the article doesn't show any real proof backing this up. They show differences between ATI's output and the reference output. But they fail to mention (and probably understand) that this is no proof of cheating. It *might* be cheating, but it's just as probable, if not more probable, that the output difference comes from the different rendering color depth (ATI 24bit, reference 32bit). Did they measure a speed difference using AntiDetector? No. So it seems to me that most likely big parts of the article's conclusions are just plain wrong.
Am I an ATI fan boy? Not at all, I've used 3dfx and NVidia until today. My next card might very well be an ATI card, though.
Re: Unbelievable IMHO
Perhaps you need to read whats being said in this thread again to understand why the testing methodology is flawed. Its all here.
chavvdarrr said:
Perhaps you need to read whats being said in this thread again to understand why the testing methodology is flawed. Its all here.
Re: Unbelievable IMHO
Well they have tried to "visualise" the difference between the images, now I don't know how they exactly did this but most likely they have ended up scaling the values. This means that a +/- 1 error can end up looking huge and ugly while in reality it is really no big deal.
A better approach is the good old trusty RMS errors, whats the average error of the whole image ? If this comes back as quite low then no worries, if its quite large then there might be an issue. For example the elephant/rhino scene looks quite noisy in their dif image, what most likely happens there is that every pixel is maybe 1 or 2 values different from the ref rast value (e.g. 151-168-154 versus 150-167-154), given the accuracy difference this should be no problem - but when using their visualisation it gives the impression of being very wrong. Another thing to look at is large per pixel errors, so if - say more than 10 pixels - have a very large error (for example 5 values RGBA or more) then again this raises some questions... but even that error might be valid if an arithmetic op ends up saturating early.
So better approach would be to do some statistics on the true error delta between ref and GF Card X/Y/Z/W, not use some weird equalize/stretch/gamma operation to try and show how every pixels is ever so slightly different.
But as said before even this approach can be quite tricky because of all the potential differences in accuracy used throughout the pipeline.
Just my 2c...
K-
chavvdarrr said:
Well they have tried to "visualise" the difference between the images, now I don't know how they exactly did this but most likely they have ended up scaling the values. This means that a +/- 1 error can end up looking huge and ugly while in reality it is really no big deal.
A better approach is the good old trusty RMS errors, whats the average error of the whole image ? If this comes back as quite low then no worries, if its quite large then there might be an issue. For example the elephant/rhino scene looks quite noisy in their dif image, what most likely happens there is that every pixel is maybe 1 or 2 values different from the ref rast value (e.g. 151-168-154 versus 150-167-154), given the accuracy difference this should be no problem - but when using their visualisation it gives the impression of being very wrong. Another thing to look at is large per pixel errors, so if - say more than 10 pixels - have a very large error (for example 5 values RGBA or more) then again this raises some questions... but even that error might be valid if an arithmetic op ends up saturating early.
So better approach would be to do some statistics on the true error delta between ref and GF Card X/Y/Z/W, not use some weird equalize/stretch/gamma operation to try and show how every pixels is ever so slightly different.
But as said before even this approach can be quite tricky because of all the potential differences in accuracy used throughout the pipeline.
Just my 2c...
K-
chavvdarr;
here is an better way to compare the different pictures :
http://www.forum-3dcenter.org/vbull...55da53&threadid=79978&perpage=20&pagenumber=9
Zeckenzack has posted an picture an an description how to compare the different pictures to get an real result. In his opinion the pics from digit-life are all flawed and useless.
NV25 on the left, R300 on the right compared with the original pictures from Xbit ( 3dMark01.nature )
You see the R300 difference is far smaller, contrary to what was posted at digit-life/Xbit
here is an better way to compare the different pictures :
http://www.forum-3dcenter.org/vbull...55da53&threadid=79978&perpage=20&pagenumber=9
Zeckenzack has posted an picture an an description how to compare the different pictures to get an real result. In his opinion the pics from digit-life are all flawed and useless.
NV25 on the left, R300 on the right compared with the original pictures from Xbit ( 3dMark01.nature )
You see the R300 difference is far smaller, contrary to what was posted at digit-life/Xbit
Yes - automatic contrast expansion of difference images seems like a very bad method.
Imagine you have two images :
Image 1 has very small differences in the low bits of quite a lot of the pixels due to inevitable variations in exact rendering as discussed above.
Image 2 also has just as many low bit errors, but also has some very large errors caused by cheats, bugs or whatever.
You take these two images, take the difference from the reference and then run the results through a contrast expander (like Photoshop). What happens?
The answer is that the results for image 1 probably look much worse, because all of the small differences in the pixels are massively magnified in the expansion operation, so it looks like the whole screen is different. On the other hand, because image 2 has some very large errors these are scaled slightly up to full brightness, and all the low-bit errors remain close to black because the overall scaling of the image is proportional, so with all the small errors in image 2 remaining practically invisible it looks like it is actually closer to the reference.
Zeckenzack shows a good example of how this works in his post. If you paste two images together into a single image before the expansion then things look very different.
Imagine you have two images :
Image 1 has very small differences in the low bits of quite a lot of the pixels due to inevitable variations in exact rendering as discussed above.
Image 2 also has just as many low bit errors, but also has some very large errors caused by cheats, bugs or whatever.
You take these two images, take the difference from the reference and then run the results through a contrast expander (like Photoshop). What happens?
The answer is that the results for image 1 probably look much worse, because all of the small differences in the pixels are massively magnified in the expansion operation, so it looks like the whole screen is different. On the other hand, because image 2 has some very large errors these are scaled slightly up to full brightness, and all the low-bit errors remain close to black because the overall scaling of the image is proportional, so with all the small errors in image 2 remaining practically invisible it looks like it is actually closer to the reference.
Zeckenzack shows a good example of how this works in his post. If you paste two images together into a single image before the expansion then things look very different.