3DMurk03 new cheats ?

Josiah said:
man this is getting insane. will we ever be able to trust benchmarks again? for now the only benches I trust are custom-made timedemos, like B3D is using.

Heh...not that bad. Just take "popular" benchmark results and multiply them x .74 for nVidia nv3x products, and multiply them x .98 for ATi R3xx products and you'll hit a general approximation of what you can expect in "real 3D games" (as nVidia likes to call them these days.)

(Just joking...sort of...:D)
 
Russ is making a good point by drawing our attention to the Quack issue and some of the proffered explanations for that. That is, it seems plausible for bugs to be involved. But if we're a bit more careful we can flesh out all the issues:

In both cases, it is obvious that the driver is making application-specific changes to various AF parameters which result in a speed-up over the default rendering path (i.e. that taken when the application recognition is defeated). Of course it should be said that from FM's point of view, any application-specific driver code constitutes an "illicit optimization" (not to be confused with...well, you know), and it is perfectly clear that Nvidia has run afoul of FM's rules from that point of view. But it does not necessarily follow that Nvidia is guilty of "the really bad thing", namely intentionally sacrificing image quality in order to get a speedup in a popular benchmark. Instead it may be plausible that Nvidia coded an application-specific optimization which was meant to cause a speedup with no harm to IQ, but, due to a bug in either hardware or software, actually does cause IQ problems.

The argument why this may have been the case with the Quack issue relies on the notion that the optimization may have been left over from the R100 driver code base, where it caused no IQ problems, but that when carried over to the R200, did cause IQ problems due to a bug or at least a "different way of doing things" in the R200 hardware. For this theory to be true, it would have to be the case that renaming Quake to Quack would cause a preformance loss on R100 hardware (although possibly a smaller performance delta than seen on R200) but have no change in IQ. Unfortunately, it seems the only evidence we have (and are likely to get) is that the Quake-recognition code may have been in R100 drivers, and that R100 at least did not display the same IQ problem that R200 did when running Quake3 with the proper name. As such, the "bug" theory remains a somewhat plausible explanation to a continuing mystery.

Lucky for us, we can test this sort of thing for the 3dMurk controversy. We already know that, just as R100 did not display the mip-map selection problem with Q3, NV2x does not display the AF problem with 3dMark03. What we haven't seen yet (to my knowledge) is whether NV2x nonetheless suffers a performance loss when renaming 3dMark to 3dMurk. If it does, it seems most likely that this problem is indeed the result of an IQ-preserving optimization made for NV2x which, due to some bug in the interaction of the optimization with NV3x hardware, causes IQ problems with NV3x. If, on the other hand, there is no performance difference between 3dMark and 3dMurk on NV2x hardware, it seems clear that the special-casing of "3dMark.exe" is not aimed at NV2x but at NV3x, which makes the conclusion that Nvidia intended an IQ-for-performance trade-off pretty inescapable.

Of course, given the timing of everything, the "inadvertant bug" explanation seems more plausible for Quack than for 3dMurk. That's because Quake3 was out in the R100 timeframe, and the "Quack drivers" were the very first set for R200; whereas 3dMark wasn't released until Nv30 was, and this is Nvidia's 4th or 5th set of NV3x drivers. But that's all circumstancial evidence, and very weak at that.

Personally, I don't believe the "inadvertant bug" explanation for either Quack or 3dMurk. But at least with 3dMurk we can easily test whether that theory is plausible or not.
 
RussSchultz said:
Himself said:
No bug in the hardware would have the bug with one filename and not another..
Ahem...

http://www.beyond3d.com/forum/viewtopic.php?p=124473#124473

I think its worthwhile to continue to investigate until we know for sure, rather than haphazardly labelling it a cheat.

I only think it's worthwhile to duplicate the results, that's it. Seems that Dave has done that, and somebody else seems to be unable to. If the results stand up, it's a cheat^H^H^H^H^Hoptimization. :) Detecting a benchmark doesn't have an alternate explanation other than wanting things to go your way in some form or another.
 
ATi initially caught the renaming issue, which was further confirmed independently by TR, and has now been confirmed again by B3d--all three acting independently of each other. Considering what has already been confirmed about the Dets and 3DMark beyond a doubt, and considering that nVidia has spun three stories about it:

(1) a driver bug
(2) FM's deliberate anti-nVidia sabotage
(3) a series of deliberate "application optimizations"

and has yet to rebut the original FM audit report in any way, shape, or form that might be called intelligent, I would definitely think that any "further investigation" would be a complete waste of time. It's obvious what this is: it's yet another cheat within the Det code relative to 3D Mark which wasn't defeated by the 330 patch because the 330 patch apparently did not change the name of the executable from what it was in 320. In fact, the only way to be quite certain it wasn't a cheat would be to demonstrate that the 3DMark executable was renamed for 330--in which case it would have been impossible for nVidia to have cheated it based on the name of the executable.

What I want to see out of nVidia is an "optimization-free pledge" concerning all benchmarks in future drivers, especially 3DMark. I mean, if nVidia wants to murder the English language and call these optimizations I guess that will be OK providing it also pledges to henceforth and forever stop optimizing its drivers for benchmarks--don't care what they call it so long as they stop doing it. We've had such an official pledge from ATi--but curiously nVidia has yet to step up to the plate on this one. Draw your own conclusions. Personally I would applaud them for such a pledge because it is the right thing to do, of course. And also because they will be stringently monitored to make sure they keep that pledge...;)
 
At this point, I believe it's safe to claim that this is a deliberate attempt by Nvidia to falsely increase scores in benchmarks.

The real excitement is about to begin (or quietly die) as we wait for both FM and Nvidia to answer to it.
 
I'll be doing some tests again on my new PC and see what I come up with. I got a GF3 but can ask a friend if I can use his GF 5200 for tests.

US
 
Ante P says this issue is gone in the new 44.61 Detonators. It might be possible that the agreement between Nvidia and Futuremark included a deal to remove all currently existing cheats from the drivers, although it is difficult to understand why Nvidia would make such a major concession.
 
Apparently the issue is cleared up in the unreleased 44.61 drivers.

Detonator 44.03 is reported to force multi-sample masking on at all times.

Does 44.61 also do this? Note that Multisample masking problems is what the directx 9.0a patch was all about.

According to Dave the rendering discrepancy only occurs when 3dmark asks for anisotropic filtering, not when the user specifies it in the control panel.

Given that 3DMark2003 is the only application that this anomoly shows up in you need to ask whether or not the program is correctly asking the driver to perform anisotropic filtering or not.

On a related note, Beyond3D is investigating whether nVidia is trying to cheat in splintercell timedemos or not. Has it occured to the investigators that the cause of the problems they are apparently observing may well lie in multi-sample masking? Don't forget splintercell already has FSAA issues for this very reason.
 
radar1200gs said:
Given that 3DMark2003 is the only application that this anomoly shows up in you need to ask whether or not the program is correctly asking the driver to perform anisotropic filtering or not.

I see. So if you rename the exe it suddenly figures out how to ask the driver properly? :oops:
 
Dave H said:
I see. So if you rename the exe it suddenly figures out how to ask the driver properly? :oops:
Of course.
Because then the silicon bugs in nv3x that only happen with exe's named 3dmark03.exe dont effect anything!
:rolleyes:
 
radar1200gs said:
According to Dave the rendering discrepancy only occurs when 3dmark asks for anisotropic filtering, not when the user specifies it in the control panel.

Given that 3DMark2003 is the only application that this anomoly shows up in you need to ask whether or not the program is correctly asking the driver to perform anisotropic filtering or not.
Application AF works fine on my 9700 Pro regardless of how the application is named. Maybe you need to ask whether the NVIDIA driver is "listening" to the application.
On a related note, Beyond3D is investigating whether nVidia is trying to cheat in splintercell timedemos or not. Has it occured to the investigators that the cause of the problems they are apparently observing may well lie in multi-sample masking? Don't forget splintercell already has FSAA issues for this very reason.
Don't forget that the screenshots where the pixel shading problems occurred had AA disabled.

You're trying too hard to find excuses, radar1200gs.

-FUDie

P.S. Do you know the difference between multisample AA and multisample masking? Apparently not, judging by your posts.
 
That's funny, I have a checkbox in my Nvidia Driver applet to turn Multi-sample Masking on or off on my #2 Machine. Must be smoking too much of something!?!?

FUDie said "Application AF works fine on my 9700 Pro regardless of how the application is named. Maybe you need to ask whether the NVIDIA driver is "listening" to the application."

Very Good point!

anyway, This seems to be a pretty specific cheat(I don't feel the need to be Politically correct) targeted at one of the things that are probably important to a person thinking about buying one of these cards. 8X AF performance. It seems quite calculating on Nvidia's part.
Isn't the great AA/AF performance one of the big reasons to buy one of these cards right now. After all, the DX9 capabilities are moot until we have a true DX9 game to play? I wonder how many FX5800U owners bought their card because of the great high res AA/AF numbers in the benchmarks? How many potential 5900U buyers were looking at the same thing when making that decision to wait and buy?

What really gets me is that Nvidia thinks they can manipulate the data and cheat it's customers and then cry foul when caught. such Hubris!
It's a Bug?, It's Futuremarks Fault? It's everyones fault but Nvidia for turning out a hunk of silicon that couldn't live up their months long hype. It's not that it wasn't competitive, It is. But it is not the killer we were led to believe. Nvidia doesn't want to be competitive, they want to dominate.
One way or another it seems...

imho the Event Timeline to date...(Caution! Speculation Alert!)
1- Nvidia is in 3dMark beta program ... everyone is happy.
2- Nvidia finds out that NV30 is less than stellar in 3dMark03.
3- Nvidia asks Futuremark to code the Bench differently to favor Nvidia.
4- Futuremark says no!
5- Nvidia drops out of Beta program(Sour Grapes) and begins Anti 3DMark campaign to discredit the Benchmark.(More sour grapes)
6- Nvidia enlists the aid of weak and misguided Web [H]asters to trumpet the cause (with promises of more New Hardware toys to play with no doubt.)
7- Nvidia adds detection and Application specific cheats to it's drivers to inflate 3DMark scores.
7.5- My suspicous mind leads me to believe they knew what they were doing and wanted to be caught, thus furthering their assertions that 3DMark03 is junk, and further discrediting Futuremark.
8- Nvidia is caught. ET breaks Story of Driver cheats in 3DMark03.
Edited: 8.5 - Cheats in driver confirmed by reputable and intelligent reviewers at Beyond 3D
9- Nvidia says bug, then clams up.
9.5- Said Web [H]aster disparages ET for breaking story, saying they're just juiced for being cut out of the first DIII preview.
10- Futuremark releases Audit Report and a patch to prevent cheats.
11 Nvidia blames Futuremark saying, basically, they're out to get us.
12- Dell makes statement saying 3DMark03 is a valid benchmark used by them to evaluate Performance.
13- Nvidia panics at thoughts of lost revenues, Threatens Lawsuit against
Futuremark for calling them a Cheat.
14- To avoid litigation, Futuremark agrees to back off on the word 'Cheat" instead calling it an "Application Specific Optimization" which is still NOT ok.
14.5- This Author buys last 9500P on shelves at local CC. Dumps his GF4 and Nvidia for Good.
15- More Nvidia cheats found, this time with 8X AF performance.
16- ??? Who knows what's next?

Have to say, It's been interesting so far! -lol- If Nvidia could get the Silicone designers to work as diligently as the Driver team has finding ways to cheat, they might be able to come up with a good Core design that doesn't need crutches to look good. Oh how the Mighty have fallen...
 
Well i don't know the difference between em and i'm not the only one. At least i ain't thread polluting like some certain zealot.
The last line aint directed at you Fudie.
There is a post here on this page that resembles that of an person trying to seem knowledgable and right but in fact don't know didlisquat and is rambling. Sheesh and i thought i knew nothing :D
BTW i liked that part about the Nv3x silicon must be designed to recognise 3dmark :D
LOL sure made my 10 minutes
 
Application specific optomization my arse. It was a cheat, and the intention of it was clear. Then again I'm one for calling a spade a spade and letting the chips fall where they may. I don't go much for the double standards of "if the person's clientel is dirt poor they're a prostitute, but if they have some dough you must call them a call girl instead". Or I could think of many other fine examples of double, urm market speak that don't quite say what they're saying...but the underlieing meaning is obvious.

And now after all this negotiating, to gloss over the truth with words that amount to damage control for nVidia, and this was going on too. It does not surprise me they would do this though. It is the sort of thing I've come to expect of big business. Remember too the tabaco companies claims that no one knew cigerette smoking was hazardous to one's health and their charges against all the scientists that reported this. Despite their claims about when people did or did not know, my father was a chemist who worked at Celenese (merged with Heurscht in the 1980s). He got assigned to some government contracted work from time to time, and one of the projects he was assigned to in the early 1970s involved the health effects of tobacco smoke.

I also had a teacher in high school who used to teach both biology and physics, who used to be a professor at Yale teaching physics and in the school of medicine. Anyhow, she used to work for Phillip Morris until she had to leave in good conscience. She so much as said they're lieing, and she knows as she worked there. She also, I seem to remember made mention to a newer less addictive cigarette and the project got canned. They very well did know how addictive their products were, she was one of the people working on this.
 
8- Nvidia is caught. ET breaks Story of Driver cheats in 3DMark03.

you forgot:

8.5 - Cheats in driver confirmed by reputable and intelligent reviewers at Beyond 3D 8)

kalbaz
 
Kalbaz said:
8- Nvidia is caught. ET breaks Story of Driver cheats in 3DMark03.

you forgot:

8.5 - Cheats in driver confirmed by reputable and intelligent reviewers at Beyond 3D 8)

kalbaz

I'm very sorry. No slight intended. An oversight on my part. :)

I'm more than willing to concede that point! The list has been edited.
 
I'm very sorry. No slight intended. An oversight on my part.

[egoboost]
np, just thought the guys here *cough* DaveBaumann *cough* need some credit now and then for their consistently high standard. and no i'm not precluding the posters here whom also provide excellent input.
[/egoboost]

anyways.... on with the thread...
 
Back
Top