Are all timedemos now invalid because of this 3DMark Fiasco?
- Thread starter Natoma
- Start date
Reverend said:
Yeah, it would be interesting the know what he think about this mess.
Anyway Rev, my comment was really regarding the absurdy bad statement the PR department sent out as damage control. Totally unprofessionel high school quality in my view and I'm sure that others in the business has been shaking their heads too. Remember the statement was not some marketing BS (where you're allowed to, well, BS some) but the only official reaction from nVidia to the rest of the industry.
Typedef Enum
Regular
Personally, I think nVidia's PR has done nothing but gone down the tubes since Brian was brought into the mix...
That might seem really odd, given nVidia's reputation when [What the hecks his name? The head PR guy that many people hated?] was fairly prominent...But I swear up and down that they've actually gotten significantly worse since he arrived...and when I say worse, I'm talking like towards_the_end_of_3dfx worse, if you catch my drift...IE save the sinking ship kinda' worse...
That might seem really odd, given nVidia's reputation when [What the hecks his name? The head PR guy that many people hated?] was fairly prominent...But I swear up and down that they've actually gotten significantly worse since he arrived...and when I say worse, I'm talking like towards_the_end_of_3dfx worse, if you catch my drift...IE save the sinking ship kinda' worse...
Diane Vanasse
EDIT: finally overcame my lazyness and looked it up
EDIT: finally overcame my lazyness and looked it up
FYI, I sent screenshots of the Splinter Cell rendering issues to NVIDIA on Thursday - no responce yet though.
I've verified that this does actually occur on the affected cards when loading the game up normally. I still wouldn't expect to see this occur normally.
The only reason I started to look at this was that in [H]'s NV35 review Brent concluded that 9800 might be faster because the shaders were tuned better to R300's architecture. This comment stuck out like a saw thumb as I already knew this was a PS1.1 title which should favour NVIDIA if anything (and they certianly should have any compatibility issues). I quizzed Brent about this and thats when he mailed me his screenshots showing the issue, that he'd witnessed during the review but didn't know if that was supposed to happen or not. Since we've verified with the developerthat this isn't supposed to, and indeed shown that it happen on NVIDIA boards < 5600, or ATI boards, I assume that Brent has/will update this review highlighting the anonomly.
I've verified that this does actually occur on the affected cards when loading the game up normally. I still wouldn't expect to see this occur normally.
The only reason I started to look at this was that in [H]'s NV35 review Brent concluded that 9800 might be faster because the shaders were tuned better to R300's architecture. This comment stuck out like a saw thumb as I already knew this was a PS1.1 title which should favour NVIDIA if anything (and they certianly should have any compatibility issues). I quizzed Brent about this and thats when he mailed me his screenshots showing the issue, that he'd witnessed during the review but didn't know if that was supposed to happen or not. Since we've verified with the developerthat this isn't supposed to, and indeed shown that it happen on NVIDIA boards < 5600, or ATI boards, I assume that Brent has/will update this review highlighting the anonomly.
hehe
http://hypothermia.gamershardware.com/nv_bs.htm
I remembered this was good for a few chuckles
I also thought Burkes comments were a little strange, especially coming from a competitor?
Pete said:
http://hypothermia.gamershardware.com/nv_bs.htm
I remembered this was good for a few chuckles
I also thought Burkes comments were a little strange, especially coming from a competitor?
Typedef Enum said:
I disagree. And it's a matter of perspective, IMO. You were quite fond of Nvidia back in '99 and '00 and so probably overlooked PR/marketing statements that might've been otherwise distasteful.
John Reynolds said:
After seeing how nVidia is dealing with this whole issue, how many days you think it's going to take from nVidia to sue Futuremark? Just read some mysterious comments from Uttar and I got a feeling that we're going to see some more imago-trashing soon. Lawsuit would be a perfect way for this.
Miksu said:
Right now nVidia can keep insisting that "Gee mom, it's only a bug" and at least some unfortunates will believe it. If they take it to court the publicity is magnified 10x and nVidia risks certain exposure--with a legal seal on top. I would be very surprised if nVidia would want to expand publicity on this matter as it would open up a can of worms, and IMO every advertisement for their drivers boasting "up to 50 percent performance increase" for the last several *years* would be open to minute investigation. nVidia surely has more to hide than it has to gain out of such a law suit, and I'll be very surprised to see them launch a suit which would be admitting that this publicity has damaged them financially. My guess is that nVidia will simply continue to defame FutureMark, talk inanely about unspecified "bugs," and simply hope the whole thing blows over. It could well be that nVidia will quietly remove any 3D Mark recognition from their next set of Dets and simply say nothing about it. If I was nVidia I would say:
"We've looked at the situation and have found several places where our drivers do recognize 3D Mark and change their rendering behavior as a result. We will be removing that 3D Mark recognition from our drivers in the next Detonator set." And let the issue lie...
But in a sense nVidia's between a rock and a hard place (of its own manufacture, certainly.) They've advertised this glowing performance difference between nv30 and nv35 and it is *essential* for nVidia that nv35 be a success and not follow nv30's fate. Yet now it would appear as if much of that performance differential comes through 3D Mark 03 benchmark cheats. So basically nVidia is damned if they fix their drivers not to cheat the benchmark (since "performance" in the bench drops ~25%), and damned if they don't remove the cheats (because at the very least people will think their drivers are extremely buggy and at worst will think that nVidia can't be trusted to enable the hardware support in its drivers it claims to enable.)
Desperation and dishonesty are often bedfellows, unfortunately.
I'm concerned that the next set of NV drivers will still use app recognition with 3dmark03. . .in order to pop up a screen that ends the app and explains why NV regrets that given the attempts at a shakedown by Futuremark (those 100's of thousands of dollars that NV refused to pay the obvious protection racket), and the intentional hostility, defamation, and blatant attempt to destroy Nvidia by Futuremark (I don't see any other way to read "it looks like they have intentionally tried to create a scenario that makes our products look bad. This is obvious. . ."). If NV is going to stick to this line of defense, it makes a lot of sense to go this route.
No, I don't believe in that line of crap, but that's the message I'm getting from NV, in some cases explicitly and in others implied, sometimes directly, and others from the obvious surrogates and apologists.
No, I don't believe in that line of crap, but that's the message I'm getting from NV, in some cases explicitly and in others implied, sometimes directly, and others from the obvious surrogates and apologists.
I would have to agree with Jon's disagreement.John Reynolds said:
Type's comments seem to single out Brian for NVIDIA's PR deterioration (which is the gist of Type's comment).
Do you know if NVIDIA PR can say what they what without any sort of consideration whatsoever?
If you are familiar with NVIDIA PR personnels' official response to indestry-related emails you send them and know them through private non-industry related matters, you may have a different opinion of PR folks.
Bigger problem.
I think the biggest problem with the 3DMark 2003 situation isn't that we can find ways to get around it but that it changes the nature of reviewing.
(For the record I am a reviewer myself and write from this perspective).
'
One of the most important functions of journalists (online and otherwise) is to act as watchdogs and to blow the whistle when we detect corporate abuse, cheating, or other improper / unethical behavior. Investigative reporting and detective work is often a part of doing our job correctly and that's as it should be.
Benchmarks are used to report on hardware performance and oftentimes to detect potential issues or problems, but the NVIDIA problem and this current discussion casts a great deal of doubt on just how much we can trust the tools we use. Unlike the ATI Quack / Quake hack which was easily detected by any user, NVIDIA's cheating was much more subtle and could've gone detected a long time if not for inquisitiveness on the part of ExtremeTech.
My point is simple: Reviewers should not have to spend our time testing and re-testing to ensure the very tools we are attempting to use remain valid, especially when the invalidity has been introduced by a third party. Companies like having their products used as benchmarks; its an immense amount of free exposure. They won't take kindly to performance hijacking by third parties or to having their products abused and twisted for the marketing purposes of another company.
The bottom line? This type of cheating only hurts the entire industry. NVIDIA's Detonator drivers have always been famed for their ability ot increase performance. NExt time they release a new set people are going to be constantly asking whether or not the drivers cheat. Meanwhile we as reviewers have to question the performance of every benchmark we run and wonder if they've been similarly hijacked, potentially going so far as to develop new testing tools just to ensure our old ones are still valid.
We shouldn't have to do that. This type of hijacking isn't fair to readers or consumers, and its not fair to us. If we can't trust manufacturers to maintain some level of fair play then no review is going to be worth the paper its printed on simply because no one will ever know if the numbers are telling the truth.
I think the biggest problem with the 3DMark 2003 situation isn't that we can find ways to get around it but that it changes the nature of reviewing.
(For the record I am a reviewer myself and write from this perspective).
'
One of the most important functions of journalists (online and otherwise) is to act as watchdogs and to blow the whistle when we detect corporate abuse, cheating, or other improper / unethical behavior. Investigative reporting and detective work is often a part of doing our job correctly and that's as it should be.
Benchmarks are used to report on hardware performance and oftentimes to detect potential issues or problems, but the NVIDIA problem and this current discussion casts a great deal of doubt on just how much we can trust the tools we use. Unlike the ATI Quack / Quake hack which was easily detected by any user, NVIDIA's cheating was much more subtle and could've gone detected a long time if not for inquisitiveness on the part of ExtremeTech.
My point is simple: Reviewers should not have to spend our time testing and re-testing to ensure the very tools we are attempting to use remain valid, especially when the invalidity has been introduced by a third party. Companies like having their products used as benchmarks; its an immense amount of free exposure. They won't take kindly to performance hijacking by third parties or to having their products abused and twisted for the marketing purposes of another company.
The bottom line? This type of cheating only hurts the entire industry. NVIDIA's Detonator drivers have always been famed for their ability ot increase performance. NExt time they release a new set people are going to be constantly asking whether or not the drivers cheat. Meanwhile we as reviewers have to question the performance of every benchmark we run and wonder if they've been similarly hijacked, potentially going so far as to develop new testing tools just to ensure our old ones are still valid.
We shouldn't have to do that. This type of hijacking isn't fair to readers or consumers, and its not fair to us. If we can't trust manufacturers to maintain some level of fair play then no review is going to be worth the paper its printed on simply because no one will ever know if the numbers are telling the truth.
Reverend said:
To be quite honest I think that is completely ridiculous, if you think Nvidia changed their driver for your demo, if on the other hand you think that they changed it for the whole game i.e. whenever those shaders are used then I might believe it otherwise to me that is incredibly arrogant to think they would care enough to do it.
However as dave stated later it is the same in the game.
What is funny is how everything that may well be a bug is now assumed to be cheating.
I know this is a stretch, but it's something I thought of today.
When reading the Tech-Report's recent graphics reviews I've noticed they comment on the FX5200's very odd behavior at the beginning of their standard Serious Sam timedemo.
I'm now wondering if it too is related to a cheat of some nature, especially since the FX5200 does lack Intellisample which all the other GfFX cards do have.
It may be a quirk, or a bug. But after what's been happening lately I am very curious now.
When reading the Tech-Report's recent graphics reviews I've noticed they comment on the FX5200's very odd behavior at the beginning of their standard Serious Sam timedemo.
I'm now wondering if it too is related to a cheat of some nature, especially since the FX5200 does lack Intellisample which all the other GfFX cards do have.
It may be a quirk, or a bug. But after what's been happening lately I am very curious now.
BTW, theres a fourth player in this game, whom people often forget. The first third are IHV, benchmark vendor, and the reviewer. Ok who's the fourth guy ? Obviously, SicroMoft who signs the drivers and puts a stamp of "Quality hardware" on it.
Obviously WHQL testing doesnt even look at issues brought up here. They just barely touch on basic DX6-DX7 functionality. And it would be near impossible for them to do something to detect any 3rd-party specific funny stuff in drivers.
But they have their stamp of approval on it, does it mean they approve Nvidias funny stuff ? Would be interesting to hear from MS on this issue.
( BTW, im very aware of the fact that microsoft does scant little functional testing on drivers they sign .. im a developer of one that gets periodically signed and oh boy what a crap has passed through there .. )
EDIT: an afterthought. Maybe they could include a timedemo of their own published games in DX driver testing and compare the output to their own refrast.
They could be making it available to public as well .. but of course, MS being the evil giant they are, nobody would trust it for benchmarking anyways.
Obviously WHQL testing doesnt even look at issues brought up here. They just barely touch on basic DX6-DX7 functionality. And it would be near impossible for them to do something to detect any 3rd-party specific funny stuff in drivers.
But they have their stamp of approval on it, does it mean they approve Nvidias funny stuff ? Would be interesting to hear from MS on this issue.
( BTW, im very aware of the fact that microsoft does scant little functional testing on drivers they sign .. im a developer of one that gets periodically signed and oh boy what a crap has passed through there ..
)EDIT: an afterthought. Maybe they could include a timedemo of their own published games in DX driver testing and compare the output to their own refrast.
They could be making it available to public as well .. but of course, MS being the evil giant they are, nobody would trust it for benchmarking anyways.
I've noticed in the benchmark Splinter Cell where there seems to be a "hole" in the water. I guess that's the best way to describe it. It kind of looks like the shots in 3dmark2001's lobby scene where bullet holes are made in the wall. Is this what you're talking about?
on a 256MB Geforce FX 5600
on a 256MB Geforce FX 5600
Similar threads
