Cheating and its implications

Status
Not open for further replies.
I hope ATI has the creativity to turn Nvidias under handed tactics into Nvidias worst nightmare!

The best way ATI can profit from this situation is not to do anything and simply let their products (software and hardware) do the talking, IMHO.
 
Re: ATI, fighting fire with fire?

Seiko said:
Now I know this is going to attract a lot of heat but as Nvidia is allowed to win the benchies due to certain and perhaps unethical optimisations what if ATI added the same optimisations to their drivers? It could be selected in the control panel and of course should be broadcast and documented in full. Although I think this may cause unintentional damage to 3DMarks credibiltity it would show all the sites two can play at that game? At the very least the sites would have to state that the optimisations aren't really going to translate into improved game framerates but hey, at least Nvidia is shown to be using the same optimisations and as they didn't allow the option to be controlled they would certainly loose a lot of credibility! Of course I'm assuming that if ATI added the same style of optimisations to their drivers we'd see a marked improvement in the scores and would probably level the playing field?

Other than that I dont see ATI with any options, at the moment Nvidia are clearly enjoying most people seeing them win the 3dmark benchies and won't care if a few specialised sites throw a little mud because of the optimsiations in place. As I hate to see a company pretty much pull the wool over it's consumer eyes I really do hope this whole affair isn't allowed to rest.

I hope ATI has the creativity to turn Nvidias under handed tactics into Nvidias worst nightmare!

I disagree with the premise that this is no big deal for nVidia. You must remember this is not an isolated, "first-time" event for them on the subject of advertising veracity dealing with their nv3x product line. Last year nVidia was hawking nv30 as a "revolution in cinematic graphics" but now the company itself is publicly stating nv30 was a "failure." [quote, unquote.] Such comments must bring shudders of ecstasy to the poor sods who bought 5800U's on advance order, such was their trust nVidia would not dare attempt to sell them a dog.

The fact is that this revelation is just one more in a long string of damaging PR gaffes the company has made over the past nine months. In that context it is far from minor, I think, because even if you make the dubious suggestion that few people will discover what's going on here, the fact is that many, many people have grown so accustomed to a relative lack of veracity from nVidia that something like this isn't even surprising. nVidia is fast building an unsavory reputation for itself gaffe by PR gaffe.

Add to this the fact that the nv35 reference cards nVidia has released thus far from the barn for "review" by selected sites are still 2-3 months away from shipping (whereas the competitor's products that are being contrasted with them are already in the channel and shipping), and the situation is not so hot for nVidia at the present time.

As for your suggestion that ATi should start cheating, too, and in the same way...oh, gosh, I hope not...;) Now, I don't care if ATi or nVidia genuinely optimize for software--bug fixes for particular games, for instance, are real and valid forms of driver optimizations. But I think we need to draw the line when drivers are deliberately written to recognize certain benchmarks and to alter their standard rendering behavior so as not to render portions of these benchmarks in order to cut down the workload simply to get better scores. That's not optimizing--that's cheating the benchmark, the reviewer, and the customer. The good thing about not cheating is that you don't have to keep looking over your shoulder, and you don't have to make up lame "Duh, it were a bug..." excuses nobody believes when you get caught--because you don't get caught because there's nothing to catch...;)

I can remember the lame Winbench ATi "Turbo" driver scandal of a few years ago, and I hammered them for it, but judging by ATi's driver releases of the last year or so it's a lesson ATi has learned and has put behind them for good. As nVidia is slowly dragged down a few pegs, we can only hope nVidia will learn its lesson as well as ATi has learned from its errors of the past. Why has ATi done so well this past year? I think it's because the company has demonstrated that in a wide number of areas it has *learned* from its previous mistakes and taken steps to correct the obvious deficiencies that at one time plagued the company. Those actions have paid off in concrete terms for ATi. Here's hoping nVidia will walk the same path from here on out. IMO.

If not, or if ATi should fall off the wagon, thank goodness for sites like B3d, ET, and a few others, who will point it out...;)
 
Re: ATI, fighting fire with fire?

WaltC said:
Seiko said:
Now I know this is going to attract a lot of heat but as Nvidia is allowed to win the benchies due to certain and perhaps unethical optimisations what if ATI added the same optimisations to their drivers? It could be selected in the control panel and of course should be broadcast and documented in full. Although I think this may cause unintentional damage to 3DMarks credibiltity it would show all the sites two can play at that game? At the very least the sites would have to state that the optimisations aren't really going to translate into improved game framerates but hey, at least Nvidia is shown to be using the same optimisations and as they didn't allow the option to be controlled they would certainly loose a lot of credibility! Of course I'm assuming that if ATI added the same style of optimisations to their drivers we'd see a marked improvement in the scores and would probably level the playing field?

Other than that I dont see ATI with any options, at the moment Nvidia are clearly enjoying most people seeing them win the 3dmark benchies and won't care if a few specialised sites throw a little mud because of the optimsiations in place. As I hate to see a company pretty much pull the wool over it's consumer eyes I really do hope this whole affair isn't allowed to rest.

I hope ATI has the creativity to turn Nvidias under handed tactics into Nvidias worst nightmare!

I disagree with the premise that this is no big deal for nVidia. You must remember this is not an isolated, "first-time" event for them on the subject of advertising veracity dealing with their nv3x product line. Last year nVidia was hawking nv30 as a "revolution in cinematic graphics" but now the company itself is publicly stating nv30 was a "failure." [quote, unquote.] Such comments must bring shudders of ecstasy to the poor sods who bought 5800U's on advance order, such was their trust nVidia would not dare attempt to sell them a dog.

The fact is that this revelation is just one more in a long string of damaging PR gaffes the company has made over the past nine months. In that context it is far from minor, I think, because even if you make the dubious suggestion that few people will discover what's going on here, the fact is that many, many people have grown so accustomed to a relative lack of veracity from nVidia that something like this isn't even surprising. nVidia is fast building an unsavory reputation for itself gaffe by PR gaffe.

Add to this the fact that the nv35 reference cards nVidia has released thus far from the barn for "review" by selected sites are still 2-3 months away from shipping (whereas the competitor's products that are being contrasted with them are already in the channel and shipping), and the situation is not so hot for nVidia at the present time.

As for your suggestion that ATi should start cheating, too, and in the same way...oh, gosh, I hope not...;) Now, I don't care if ATi or nVidia genuinely optimize for software--bug fixes for particular games, for instance, are real and valid forms of driver optimizations. But I think we need to draw the line when drivers are deliberately written to recognize certain benchmarks and to alter their standard rendering behavior so as not to render portions of these benchmarks in order to cut down the workload simply to get better scores. That's not optimizing--that's cheating the benchmark, the reviewer, and the customer. The good thing about not cheating is that you don't have to keep looking over your shoulder, and you don't have to make up lame "Duh, it were a bug..." excuses nobody believes when you get caught--because you don't get caught because there's nothing to catch...;)

I can remember the lame Winbench ATi "Turbo" driver scandal of a few years ago, and I hammered them for it, but judging by ATi's driver releases of the last year or so it's a lesson ATi has learned and has put behind them for good. As nVidia is slowly dragged down a few pegs, we can only hope nVidia will learn its lesson as well as ATi has learned from its errors of the past. Why has ATi done so well this past year? I think it's because the company has demonstrated that in a wide number of areas it has *learned* from its previous mistakes and taken steps to correct the obvious deficiencies that at one time plagued the company. Those actions have paid off in concrete terms for ATi. Here's hoping nVidia will walk the same path from here on out. IMO.

If not, or if ATi should fall off the wagon, thank goodness for sites like B3d, ET, and a few others, who will point it out...;)

I think in a perfectly just world I'd back that viewpoint WaltC however I honestly believe the bean counters of Nvidia have done their math. The fact that only a few sites have even mentioned this (and I believe the Nvidia PR department and board) would have considered this fact means in short they will benefit tremendously from the chart topping scores they are currently enjoying. Do you honestly think one of their sales managers want be able to convince the OEMs that it's just a simple bug. And that's only if the OEMs are even aware of the rendering path issue. Then take the JoeBloggs, again the vast majority won't even consider the rendering path issue. At the end of the day the current review sites have created a monster, "King of the Benchies" and Nvidia being well aware of that are not going to settle for their flag ship product to be shown loosing them. At the end of the day, at the very least it's a simple time delay tactic until allowing them to be on poll position until they can manage to get the drivers good enough to stay there ethically. This puts companies like ATI and no doubt many others in an awful position. Let the competition cheat or use unethical and perhaps misleading optimisations and rake in the revenue garnered from the review sites putting them on top of the benchies or should they get a little dirty too? Sure ATI can currently sit back having had the brilliant R300 but the smaller companies who rely on a fair shake of the stick and don't have multimillions in reserve, what for them?
Of course this could be taken from out of the companies hands altogether if review sites would turn both ethical and professional but with so many freebies and pressures to be the first to post we're not going to see that any time soon. After all a couple of teenagers masquerading as trained technicians or professional journalists is a ridiculous start anyway. It’s easy to give these sites initial credibility because they’re there. It’s only after a long time of viewing many sites will you be able to determine the quality of their review. Even then with the likes of Toms and Anands being taken in still not raising the issue just what level of site do we require?

Until then Nvidia and I'm sure many others will continue to play the game. Nvidia is I'm sure in no mood to play fair when money comes into it and having lost so much performance credibility with the NV30 farce I think they knew they must win practically every round to topple the R350s stranglehold on the high end performance. And in all honesty, I think people really need to wake up to that fact, Nvidia will say anything you want to hear so long as they get your purchase!

Alas though this has already pretty much fizzled out to nothing with none of the big guns really joining up and taking a stance. Sure B3D and Extreme have started the ball rolling but without the conviction to get the others on board and broadcasting the same message haven’t managed to change a thing. It’s only when the majority of sites start broadcasting the message will the companies like Nvidia and ATI really play by the rules that we the consumer set out through use of review sites and their contents. Not only will a united review site, process and professional stance ensure the companies play ball but it will also stop a lot of sites rushing absolute crud out the door for us poor consumer to swallow! Will this or even can this happen? With B3Ds and Extreme Techs initial guide and a lot of effort and communication between them and the other sites perhaps? Without their follow up, communications to fellow sites and passion for the truth to come out, not a chance. My bet, Dave and co. simply won’t have the energy or perhaps desire to actually change the current review sites process so in short I'd score this 1 to the Nvidia bean counters and 0 to the public, competition and reviewers.

:(
 
Err, how quickly are you expecting people to act on this? This issue is just beginning.

Also, Tech TV was mentioned at some point in connection to this...have they mentioned it, or was that speculation that they might?
 
demalion said:
Err, how quickly are you expecting people to act on this? This issue is just beginning.

Also, Tech TV was mentioned at some point in connection to this...have they mentioned it, or was that speculation that they might?

I would expect sites to address the fact that their findings may be inaccurate as soon as possible! Everysite that has published any 3Dmark scores should in a perfect wolrd all now be printing that a) their findings may be inaccurate and should be taken with a grain of salt for the time being, b) they are currently investigating the issue and will respond when further information comes to light c) what are the consequences of the findings are proven to be incorrect/correct.

Again, I know I'm expecting too much but at the end of the day review sites really need to make an effort and step up a level!
 
I am not too worried as I have so many friends that I will tell NOT to buy nVidia as their hardware is NOT up to the task and that they have to resort to cheating in their drivers to compete.
 
Seiko,

In your response to me above, I see your point--but I guess I'm not quite as cynical...;)

nVidia didn't anticipate getting caught, remember, so it's doubtful the bean-counters ever anticipated the scenario--indeed, it's doubtful the bean counters ever knew anything about the driver hacks relative to the benchmark(s?). Telling people "it's a bug" is certainly not indicative of a well-thought strategy which was already in place for the eventuality of getting caught...;) In fact, it indicates to me that it was a total knee-jerk response as it is so transparent.

I think you are making nVidia out to be far too monolithic. Companies like this are departmentalized and often the departments communicate with each only through middle managers. Often the PR department has no idea of what's going on in the software driver department. I would imagine the pressure at nVidia for the nv35 pre-shipping debut was nothing short of incredible--given that even in nVidia's eyes (let alone the general market) nv30 was a "failure." I think if anything you may be underestimating the damage done to nVidia's reputation by the nv30-R300 face-off. IMO, nVidia made with nv30 exactly the same kinds of mistakes 3dfx made with the V5--they hyped the crap out of it months before they even knew if they could make it and ship it in quantity, and long before they knew about clock rates and yields.

The point is that pressure for nVidia with nv35 is intense, as is the pressure that it "look good" relative to its R350 competition. The point is that *somebody* high enough up in the nVidia hierarchy to make such decisions felt the nv35 products needed this kind of driver hack to present a competitive appearance in the general marketplace. They also felt the need to present a specially prepared made-for-and-by-nVidia Doom III demo, released only to sites guaranteed not to analyze it too closely (who may have in fact had to agree not to analyze it too closely), to attempt another kind of warped performance-comparison picture. To do what, really?

Not to sell the $500 nv35-based products they released temporarily to a few sites because those are yet months away from retail shipment. So the goal here was obviously to slow ATi's market momentum (since R3xx products are obviously shipping) by having people put off ATi purchases they may have planned. The expose' has thrown a great big monkey wrench into the works--ie, the plan of somebody at nVidia (which was to cheat and *not* get caught) has backfired--and really in kind of a double-whammy manner, because now more than ever there is also a cloud hanging over the "fairness and objectivity" of the made-by-and-for nVidia Doom III demo.

Anyway, I can partially agree with your sentiments concerning system OEMs in that "they don't care." I think they don't care particularly whether its ATi or nVidia or whomever's products they sell. What they care about is customer demand, though. And this is exactly the type of thing that can cause customers to choose one product over another when placing orders for their systems.

Last, what kind of position is nVidia in right now because of this? Not a good one, assuredly. Because of the nature of what's been done, people will expect to see performance drop when nVidia "fixes" its drivers (which it will now have to do since it has already said that their driver has a "bug" which has caused this problem.) If performance doesn't drop it will raise eyebrows again and people will suspect that the driver is still cheating but in a different way (by changing rendering modes in 3D Mark to normal only when the camera goes off track, for instance, and changing it back to cheat mode when the camera track is normal, or something similar.) nVidia has painted itself into an uncomfortable corner, it seems to me.

Last, the worst thing ATi could ever do would be to adopt similar cheats, IMO. Their position is much stronger as it is. As it is only nVidia is cheating in this manner--the Catalysts pass the same test the Detonators fail. Why would ATi want to be in the same boat?
 
waltc, since JC from ID was a part of the plan to deceive people, I have decided to reconsider buying doom iii. JC tried to save face at the last minute and failed. If he truly was impartial and a desent Guy he would have let ATi know and provided them with the same oppertunity...

I think ATi will definately have the last laugh here as HL 2 is looking very sweet now...

Personally I think Kyle from HardOCP is a slimy guy, while other sites choose to obmitt driver results that are erronious, Kyle post them anyway. You can be rest assured that if the situation was reversed, he would not have posted the nV35 results with the newer driver....

Nvidia if you are reading these postings, I STRONGLY SUGGEST that you focus on producing quality hardware instead of driver hacks....
 
YeuEmMaiMai said:
waltc, since JC from ID was a part of the plan to deceive people, I have decided to reconsider buying doom iii. JC tried to save face at the last minute and failed. If he truly was impartial and a desent Guy he would have let ATi know and provided them with the same oppertunity...

Hang on I missed something crucial.... JC was a party to deceiving people regarding performance in Doom III and NVIDIA cards??
 
It is too bad it takes so much work to hand-tune such a cheat driver (which, in my opinion, should be getting bigger play than it is --these are hours of valuable developer time *not spent making the drivers better*). Otherwise I would suggest ATI do a clearly-labeled one-time "NV35 Homage Driver" with the exact same techniques just to show the real-world impact. Of course, if they listen to Kyle they will do so, but not label it anything and just leave it in for all subsequent versions. . .
 
yes and this is so

1. when people inquire about benching D3, he referrs them to nVidia.

2. ID did not notify ATi of the comparison between video cards.

3. He tried to save face by changing the demo at the last minute.


Tahir said:
YeuEmMaiMai said:
waltc, since JC from ID was a part of the plan to deceive people, I have decided to reconsider buying doom iii. JC tried to save face at the last minute and failed. If he truly was impartial and a desent Guy he would have let ATi know and provided them with the same oppertunity...

Hang on I missed something crucial.... JC was a party to deceiving people regarding performance in Doom III and NVIDIA cards??
 
In answer to number 2. I think JC is still p'd off at ATI for leaking the Alpha build.

3. I dont know what you mean, please elaborate.


Finally number 1 has already been discussed.....refer to Reverend for an answer there ;)
 
Nvidia came to the party with their own demo recorded for doom 3.

Tahir said:
In answer to number 2. I think JC is still p'd off at ATI for leaking the Alpha build.

3. I dont know what you mean, please elaborate.

finally ATI did not leak the demo and the so called Memo from JC was a fake.


Finally number 1 has already been discussed.....refer to Reverend for an answer there ;)
 
YeuEmMaiMai said:
yes and this is so

1. when people inquire about benching D3, he referrs them to nVidia.
Can you tell me where JC actually/personally said this?

2. ID did not notify ATi of the comparison between video cards.
Probably true but I fail to see where id felt this was of importance (and if you're thinking about the whole "possible cheating" thing, I'm not sure id's position is that IHVs always try to cheat... I don't think they'd like it if they knew they're being taken advantage of by an IHV... just my opinion of course) -- NVIDIA probably arranged the whole thing from arrangement to actual benchmarking and I personally think security issues is the most important thing id was worrying about, instead of "fair chances". Perhaps the "fault" is just as much as ATi that they didn't arrange for this (or maybe they did try but didn't succeed... dunno really... I hate speculating). Oh well.

3. He tried to save face by changing the demo at the last minute.
JC "saving face". LOL.
 
Reverend said:
2. ID did not notify ATi of the comparison between video cards.
Probably true but I fail to see where id felt this was of importance ...

Maybe, but in that case I would say that Id were surprisingly ignorant about the impact their benchmark would have on something that might not have been a level playfield (in regard to how mature the drivers from different IHV's were at this point).

Reverend said:
NVIDIA probably arranged the whole thing from arrangement to actual benchmarking and I personally think security issues is the most important thing id was worrying about, instead of "fair chances".

You're probably right and if the drivers from the different IHV at the point of publication were at level playground (John Carmack would know this) I don't care who arranged what and when.

I actually think that it would be a decent gesture from John Carmack if he would update his plan to whether his performance estimate from January are still true (R300 and NV30 more or less equal) or whether the latest benchmark numbers are true. :|
 
Tahir said:
I hope ATI has the creativity to turn Nvidias under handed tactics into Nvidias worst nightmare!

The best way ATI can profit from this situation is not to do anything and simply let their products (software and hardware) do the talking, IMHO.
I agree 100%
 
OK here you go straight from your own forums

Beyond3D asked ID if they could get such a benchmarking opportunity and this is how it went:

B3D:"Is there any way we could work together on a similar basis as per what was accorded to HardOCP and AnandTech?"

What they replied was :

ID:"Anthony,

NVIDIA chose those outlets - we did not. It would be best to place this
request with them."

So ID (John Carmak is ID) is telling you that you will need to go through Nvidia to get a appointment with ID software.

http://www.beyond3d.com/forum/viewt...t=5776&start=20

I am not totally stupid here and yes he did try to save face by chaning the demo at the last minute....

Reverend said:
YeuEmMaiMai said:
yes and this is so

1. when people inquire about benching D3, he referrs them to nVidia.
Can you tell me where JC actually/personally said this?

2. ID did not notify ATi of the comparison between video cards.
Probably true but I fail to see where id felt this was of importance (and if you're thinking about the whole "possible cheating" thing, I'm not sure id's position is that IHVs always try to cheat... I don't think they'd like it if they knew they're being taken advantage of by an IHV... just my opinion of course) -- NVIDIA probably arranged the whole thing from arrangement to actual benchmarking and I personally think security issues is the most important thing id was worrying about, instead of "fair chances". Perhaps the "fault" is just as much as ATi that they didn't arrange for this (or maybe they did try but didn't succeed... dunno really... I hate speculating). Oh well.

3. He tried to save face by changing the demo at the last minute.
JC "saving face". LOL.
 
YeuEmMaiMai said:
"Is there any way we could work together on a similar basis as per what was accorded to HardOCP and AnandTech?"
...
So ID (John Carmak is ID) is telling you that you will need to go through Nvidia to get a appointment with ID software.

How exactly does what Rev is asking morph into your conclusion?

Answer? It doesn't. 'work together on a similar basis as per what was accorded to HardOCP', which included having an NVIDIA guy fly a rig out with hardware, etc (or was it to pay to get a reporter from the outlet to iDs studios) and organize an event. This does not equate to "an appointment with iD software".
 
1. hardocp stated that they tested on their own equipment. (you did read that part right?)

so B3B asks if they can bring in their own equipment and run some benchmarks and they are told to go talk to nVidia??


please use some common sense here.


ID does not need to refer anyone to nVidia in order to test some hardware on a doom 3 benchmark using the build that was available to nVidia.
 
An interesting caveat to the story is that the PC that was to be delivered to us here at [H] on Friday for testing was lost by the airlines and never recovered anywhere close to where it was needed. Jim Black, in NVIDIA developer relations, was keen enough to carry a duplicate hard drive on his person while attempting to deliver the system to us.

To make an excruciatingly long story...not so long, we built a machine here to test DOOM 3 on. However, due to strenuous hardware security measures taken by id and NVIDIA to protect the DOOM 3 data from leaks, we were not able to use the scripted tools that were to make our benchmarking much speedier. To outline the facts we: A. Benchmarked on hardware put together and owned by HardOCP.

Here, we have text suggesting that NVIDIA was delivering the system, but something went wrong and HardOCP ended up supplying the system.

At Anand:

We were given one evening with a system running Doom3, to test the latest GPUs
Is that them saying "we were allowed an evening with a system (that we provided)", or is that them saying "we were allowed an evening with (access to) a system"

I don't know what Tom had to say about it.

Either way, its a bit ambiguous at best and there's more to what went on (in terms of logistics) than simply "being allowed to benchmark".
 
Status
Not open for further replies.
Back
Top