Will Tom's, Anand, and other sites carry the 3DMark story?

[OICAspork]

I checked both Anandtech and Tom's Hardware... so far I found no reference to the 3DMark cheat on Anandtech, even on their news page. On the other hand, I did find a sort of tip-toed entry on the news page of Tom's that was basically of the type "We aren't really commenting on this... go here and read about it", which was fine for a news type post, but is it likely that either site will run with the story on their front page... or will NVidia flex its muscle and coerce them to refrain from doing so? I think both of these sites have enough influence on their own to be able to stand up to NVidia with little to know negative reprocussions from NV... but I don't know that either of them will. I do think that if either of these big sites do decide to post the story, they will allow NVidia a chance to respond before posting it, which would explain why they haven't posted such a story yet.

Does anyone have thoughts on this?

Also... I wonder if Kyle over at [H] will swallow some crow and post an apology. I hope so.

Woah... ^^/ when I went to submit I noticed the poll option... this seems like an appropriate time to use it. Please check all that you think apply.

~Edit~ Tom's has since submitted a bit more on the subject...

What does this mean in the big scope of things? Many questions remain, but as long as reviewers as well as OEMs continue to use 3DMark03 as one of the standard tests that is used to judge the performance of hardware, these issues are going to continued to get heated. Right now, Lars and Tom are hard at work evaluating the situation and hope to provide a more in depth report shortly. While we all have our early opinions of the situation, it is clear that you, the readers want us to properly research and look at all sides of the issue before giving you our in depth opinion, on what can now only be called a mess. This is what we are going to do.

The whole current right up can be found here:
http://www.tomshardware.com/technews/20030523_192553.html

I've also heard that [H]ard had a 'it doesn't matter because 3dMark is crap' type article up, but then had it pulled due to lots of angry people in the bboard... ^^; So I don't think there will be an apology to Extreme Tech.

 

[OICAspork]

Oh, here is a link to the forum discussion where Tom's is actively seeking input on what to include in the article they are working on the subject... so please contribute your thoughts here:

http://www.community.tomshardware.c...493999&page=0&view=collapsed&sb=5&part=1&vc=1

Come on, tell Tom's what you want to see in the article... and hope they listen!

 

[Joe DeFuria]

My choice isn't on there.

What I think will happen, sadly, is what happened at HotHardware. Despite the fact that FutureMark provided a mountain of evidence, and clearly outlined the case against nVidia, and despite the fact that nVidia has no response to the evidence, just unsupportable claims of purposeful wrongdoing by FutureMark, the sites will treat this as a "we can't really trust EITHER of them.

So they'll post benchmarks from both Pre and Post patch 330....not to show the difference or illustrate the effects of cheating, but to say "OK reader, you decide. We don't know who to believe."

Problem is, nVidia hasn't, at least yet, made ANY CASE for us to believe "their side" of the story. They haven't GIVEN us their side of the story, despite having been informed over a week ago.

Sad...utterly sad.

I expect Anand, who doesn't use 3DMark at all, to just completely ignore it. Despite the fact that it's a hugely newsworthy story in and of itself.

Anand is supposed to have a "detonator comparison" or a "new detonator" review or something like that...and it's overdue. I have a hard time believing he just wouldn't say ANYTHING about it. Mu hunch is something like this: "We've heard something about several vendors cheating on 3DMark, which we don't use. That doesn't have any impact on us, so we'll just get on with it."

Sigh...

 

[Tagrineth]

First off, for once, Tom seems to be doing something seriously right. Bravo. Let's hope his conclusions are logical this time, as well. I remember Tom was one of the reviewers who tried desperately to show GeForceFX in teh best light possible (nVidia wins back the crown!)... will he turn around and admit something's up? Or will he defend nVidia like he has been for far too long now?

Anand is supposed to have a "detonator comparison" or a "new detonator" review or something like that...and it's overdue. I have a hard time believing he just wouldn't say ANYTHING about it. Mu hunch is something like this: "We've heard something about several vendors cheating on 3DMark, which we don't use. That doesn't have any impact on us, so we'll just get on with it."

For some reason... I get the odd feeling that Anand may have bit off more than he could chew in his intial timeframe, with the suspicion of cheating... he could be scrutinising all of his vastly differing results between drivers, trying to see if they're visibly cheating in anything else. Maybe. Either that or he's dead. =)

 

[Joe DeFuria]

First off, for once, Tom seems to be doing something seriously right. Bravo.

Agreed. One thing that Tom's does like to do is "expose evil companies" (justifiably or not.) If this isn't an opportunity to do that, there never will be one.

For some reason... I get the odd feeling that Anand may have bit off more than he could chew in his intial timeframe, with the suspicion of cheating... he could be scrutinising all of his vastly differing results between drivers...

One can only hope!

 

[Doomtrooper]

I don't go there, too many politics and a certain Fan Site Webmaster doing video card reviews.

 

[Pete]

Is Anand going through finals now, or is that over? That's the only thing I can think of that would delay his response to this issue so greatly. Or maybe he's just being incredibly thorough (checking his questionable Q3 & SC scores, confirming with id that the D3 numbers are accurate, talking with nV about the 3DM03 "fiasco"). I think he's saving comment for his Det FX article, which he must have expanded since these "issues" came to light. Evan has at least said they're getting to it in the comments thread on Anand's 5900U article (after repeated prodding--yeesh).

I just hope he's not holding back for fear of offending nVidia, when clearly they're in the wrong. Evan's reply seems to indicate no, but I'll have to see what Anand's Det FX article says to be sure.

 

[jjayb]

Or maybe he's just being incredibly thorough (checking his questionable Q3 & SC scores,

Pete, I'm begining to wonder if they are ever going to get around to that. Since they posted on the board about looking into it we haven't heard anything. That was what? 3 or 4 days ago? How long does it take to check and make sure AA is on? How long does it take to remove the statement about only ATI drivers having a problem with AA and put that all cards that use Multisample AA have this issue? I still think it's pretty ridiculous that this bad information has been left uncorrected in this article more than a week later.

 

[OICAspork]

I don't go there, too many politics and a certain Fan Site Webmaster doing video card reviews.

Yeah, I was really hard on Lars after he posted that NVidia PR BS refuting 3DMark03's legitmacy... I joined the forums that day and railed on him... and to be honest I think my forum entry... that snowballed is a good chunk of the reason that the follow up article was written (He said that he had always intended to do the follow up, but there was no suggestion that a follow up was in the works in the first article that just regurgitated NVidia's party line). That said, his follow up article was pretty good, and I really think that he is getting so sick and tired of NVidia's crap... I mean they intentionally led him on about how the NV30 was so superior to the R300 (suggesting things like an 8x2 architecture)... that I think he is starting to finally come around and hopefully is honestly becoming less biased.

 

[jjayb]

You have to love this one. It's none of the above mentioned sites. It's a gamer's site called shacknews. www.shacknews.com

I cut and pasted their blurb:

A while back many of you may recall ATI getting caught red-handed having their drivers detect Quake3 firing up and they would drop the image quality slightly to improve benchmark scores. That was just a tiny bit of a scandal. Well now Futuremark is accusing NVidia of playing naughty when it comes to their benchmark suite:

"Recently, there have been questions and some confusion regarding 3DMark 03 results obtained with certain Nvidia" products, Futuremark said in the statement. "We have now established that Nvidia's Detonator FX drivers contain certain detection mechanisms that cause an artificially high score when using 3DMark 03."

The real question being, is NVidia dropping image quality to inflate those scores? If NVidia is simply adjusting how their driver reacts to the application to improve performance without hindering the quality, then I dont see the problem. Just about all hardware vendors do this to make sure their stuff works with the most popular game engines. Also keep in mind that NVidia (among others) has been vocal in the past about the latest 3DMark not being a very good benchmark, that could play a part in Futuremark not exactly being thrilled with NVidia and perhaps trying throw some bad mojo their way. update: It seems that ATI is also being looked at and I'm just blind (Thanks Rick):

Our investigations reveal that some drivers from ATI also produce a slightly lower total score on this new build of 3DMark03. The drop in performance on the same test system with a Radeon 9800 Pro using the Catalyst 3.4 drivers is 1.9%. This performance drop is almost entirely due to 8.2% difference in the game test 4 result, which means that the test was also detected and somehow altered by the ATI drivers. We are currently investigating this further.

Will the insanity never end?

 

[OICAspork]

You have to love this one. It's none of the above mentioned sites. It's a gamer's site called shacknews. www.shacknews.com

Will the insanity never end?

I weep for that website... seriously there are more NV fanboys there than in NVnews... Shack news IS NOT supposed to be an NVidia fansite is it? o.o; It really is baffling then that the majority (not a large minority, the majority) are saying that it doesn't matter because Futuremark/Synthetic benchmarks are worthless... brrr... scary stuff!

 

[3dcgi]

The real question being, is NVidia dropping image quality to inflate those scores? If NVidia is simply adjusting how their driver reacts to the application to improve performance without hindering the quality, then I dont see the problem.

Comments like this clipped from the Shacknews quote above sound good when you don't know the whole story. In general I'd agree with that statement, but as the 3d mark situation shows this specific optimization is not possible in a real game and thus it is a cheat and not an optimization. Reading comments like that makes me think the author doesn't understand the technical details of the "optimization."

 

[Reverend]

I am going to be both diplomatic and polite with my comments about the different reactions and views on this entire matter from various websites/media.

I think the problem here for the "uncertain" sites (those sitting on the fence, so to speak) is that they are not FM beta members and hence do not have access to the developer build that allows the freecam mode, which in turn allows a person to see/experience first hand the rendering anomalies that occurs with the NVIDIA drivers and pre-330build of 3DMark03. The usual phrase of "Until I see it myself, I really can't be sure" applies here I think. Of course, B3D, ET and FM have provided the screenshots and regardless of any respect any website may have for the three, I think we shouldn't dismiss the fact that these websites :

a) are incredulous that this is what NVIDIA is doing, and the extent to which they are doing it as revealed by FM's audit report
b) are competitors with each other
c) a bit miffed that they can't experience this first hand (as opposed to simply being "jealous" they're not a website on FMs beta member list... I doubt this is the case)

It is easy for B3D and other beta members to have a very definite opinion on this matter (because we have hands-on experience) and it is easy for the public that trusts B3D or ET or FM to also form a definite opinion similar to B3D/ET/FMs due to their trust/respect. I am not so sure about the rest.

Personally I can understand the uncertain stand of various parties. They simply haven't been able to experience this first hand. When you do, I think it makes forming a definite opinion about this all that mush easier, especially when you have so many different hardware to compare against NVIDIA's.

My personal opinion, of course.

 

[jjayb]

Maybe Futuremark should release an .AVI or .mpg of the going off the rail cheat in action. That way people could see how obvious the cheat is.

 

[OICAspork]

Wow Reverend... that was very well put, and very reasonable. :/ It would be incredibly frustrating for a hardware site that considers themselves to be 'tier-1' to not have the ability to recreate/verify/disprove FutureMark's claims due to lack of access to the BETA partner free cam mode. I really don't see any work around either, other than those screens. Maybe Futuremark releasing a video of the footage in free cam mode demonstrating the cheats would help, but still it wouldn't allow for independent verification. Thanks for sharing your thoughts, Rev!

~edit~ D'oh! jjayb beat me to the video idea!

 

[Pete]

I don't know, Rev, IQ differences have been pretty visible before, so it's been obvious nVidia has been doing something not kosher with 3DM03 for a while.

OTOH, as much as I've been following the public evolution of the NV30 & NV35, even I still don't know exactly why those two cards can't seem to perform at least comparably to the R300/350 in pure shader benches (though I did tend to skim over the more technical threads). I can see how some more general hardware sites may not be as up to speed with the technical aspects of both cards, and thus still tend to believe that drivers are impairing the FX, not the hardware itself.

I expect all sites to comment on this in an article, including Kyle. Tom is obviously polling for guidance, Anand will probably include this in his Det FX review, and Kyle will put up an article somehow, perhaps as an addendum to his previous Benchmarking one.

 

[cho]

Anandtech? they have never use the 3dmark03 to do graphics benchmark in my memory.

 

[Lezmaka]

I think Kyle is pissed because he isn't part of the beta program and is doing whatever he can to try and invalidate FM and 3DMark, especially if that means he gets to agree with nvidia.

Now that makes a lot more sense than FM trying to get back at nvidia, or whatever Kyle's saying.

lol

 

[Sage]

man at the Dec 2002 [H] workshop there were several things that were supposed to be given away but were "forgotten" namely some BestBuy gift certificates. They said that they would give them out later and mail them. Well looooong time later what happens? They are giving away the "leftover" BestBuy gift certificates on the website *to anyone who registered for the drawing online* which REALLYpissed me off because, considering you had to preregister to attend, they should have no prob holding a drawing *for only the people that attended.* So, why did they give them away online? FOR THE GODDAMNED HITS!!! BB donated those gift certificates to be given to the people at the workshop but then the [H] turns around and uses them to get more hits, thus make more money. I now see what the [H]'s real motives are...

 

[no_way]

CNET News story indicates its pretty serious.

Edit: Ah, from Firingsquad i read this

NVIDIA said that a bug in the Detonator 44.03 driver is the cause of this issue, but it may be a little more than that.

Has anyone done tests with different detonators ( leaked betas etc ) yet ? If other drivers show similar tendencies this claim can be officially debunked.[/quote][/url]