New 3DMark03 Patch 330

[Tim]

This has been addressed in this thread already.

The problem is, this isn't a game we're dealing with. It's a synthetic benchmark. Coming up with fine-tuned "optimized" shader code like that isn't useful here, because the performance boost from them cann't be translated to real-world performance gains as it can from a game's timedemo to the actual game.

No it has not. This quote only addresses the issue of replacing shadercode and it does not contradict what I wrote. Replacing code is not rendering what the developer intended even if it looks exactly the same and I agree that would be cheating especially in synthetic benchmarks.

 

[Bjorn]

No it has not. This quote only addresses the issue of replacing shadercode and it does not contradict what I wrote. Replacing code is not rendering what the developer intended even if it looks exactly the same and I agree that would be cheating especially in synthetic benchmarks.

But you said:

There are no image quality degradation with Ati´s drivers and they have no problems with the free lock mode. Other than the performance improvement there seems to be no difference, this indicates that it is optimizations not cheats. (Ati could off cause be using some kind of free mode detection).

 

[FiggyG]

This has been addressed in this thread already.

The problem is, this isn't a game we're dealing with. It's a synthetic benchmark. Coming up with fine-tuned "optimized" shader code like that isn't useful here, because the performance boost from them cann't be translated to real-world performance gains as it can from a game's timedemo to the actual game.

No it has not. This quote only addresses the issue of replacing shadercode and it does not contradict what I wrote. Replacing code is not rendering what the developer intended even if it looks exactly the same and I agree that would be cheating especially in synthetic benchmarks.

But Futuremark does not agree

Any changes to the way it renders is cheating.

 

[Bjorn]

Replacing code is not rendering what the developer intended even if it looks exactly the same and I agree that would be cheating especially in synthetic benchmarks.

I actually disagree. If it looks exactly the same, then you're rendering exactly what the developer intended, or aren't you ?

But as has been mentioned already, this is not the problem in this case.

 

[Myrmecophagavir]

HardOCP has updated its news post:

Futuremark has released a patch for 3DMark 2003 that eliminates “artificially high scoresâ€￾ for people using NVIDIA Detonator FX drivers. This is in response to the news item we posted last week ( as did several sites ). According to the PDF on Futuremark's site, the patch causes a 24.1% drop in score for NVIDIA and a 1.9% drop in score for ATi meaning NVIDIA isn't alone in this whole ordeal, other companies have "irregularities" as well. It is good to see Futuremark fix the current situation to remove any unfair advantages, but this goes to show you what we meant in our "Benchmarking Right" article. Thanks to everyone who sent this one in.

I think people jumped on the site too quickly; the initial wording did come across in a bad way but it didn't appear to me to be laying blame, rather reinforcing their POV that the benchmark scores may be invalid. Of course the cheats affect individual test scores as well as the overall score, so it didn't make sense to single that out, and the issue will affect game benchmarking too since the shaders used there will be just as recognisable.

No it has not. This quote only addresses the issue of replacing shadercode and it does not contradict what I wrote. Replacing code is not rendering what the developer intended even if it looks exactly the same and I agree that would be cheating especially in synthetic benchmarks.

If you replace the code such that there is no difference in output, then yes you are still rendering what the programmer intended. However you are not necessarily using the method intended. It's debatable whether this would be good in games - the better solution is to contact the developer and make sure the original game has a more optimal shader. However for games that are no longer being patched with updates, the only way to produce this optimisation would be to detect it in the drivers. But this argument doesn't count for a benchmark test - the purpose of that is to see how fast the card is running predefined code. If the code is not the same from card to card no comparison can be made (or rather, a comparison of ability to produce desired effect can be made, but not of speed at specific algorithms). If ATI is replacing the code (which hasn't been proved yet it would seem) with something that produces identical output but is more speed-optimal, it invalidates the tool as a benchmark.

 

[Nite_Hawk]

Bjorn:

Well, that begs the question. If it's the *exact* same output, you should submit it to futuremark as an enhancement to the code. This is why it makes sense for board/chip producers to be beta members. If they arn't willing to look at it, then you should release a set of drivers that specifically points this out, that you can, through the use of optimzations, increase the rendering speed by x percent through some code replacement. If your open and honest about everything, it would be futuremark that looks bad for releasing unoptimized code, and you'd look good for being able to run the benchmark (even if it's with "cheating") drivers because you were honest about everything.

Edit: I should point out that any drivers with code replacement done shouldn't be the "normal" drivers that people run, but one specifically used to show that through code replacement/etc the speed of the benchmark can be increased by a certain amount.

Nite_Hawk

 

[Gunhead]

Why am I not surprised. I'm also feeling pretty good about not going there to read it.

Well, "Posted by Steve" may explain their front page blunder

(I do wonder how much Kyle is really involved there -- how well Steve represents site boss Kyle's opinions? I don't know their relationship.)

Anyway, the [H] forum on this issue is actually quite a refreshing read: the supporters of the site's apparent policy are a very small minority. Unless of course it's all tourists from B3D

[Edited typos.]

 

[Bjorn]

Bjorn:

Well, that begs the question. If it's the *exact* same output, you should submit it to futuremark as an enhancement to the code. ...

The problem is then, does this benefit all other IHV's ?

If not, then they have to make different codepath's for the other IHV's. And then you would end up with 3D Mark 2003 - Nvidia, 3D Mark 2003 - Ati, ...

 

[Calavaro]

Futuremark. I must extend my thanks to you for a job well done (corny but true). Suomalainen sisu

About the [H] reaction.
I don't think Steve really intended it to come out as it did in it's original wording. The "revised" post I think communicated the point much better.

Brent, I know you must be doing some serious tip toeing around. Not to lose objectivity and not stepping on the bosses feet, eh...

Nvidia cheated, got caught. End of round 1 I suppose. I hope everyone (consumers and reviewers alike) will keep on nvidias behind to prevent this sort of cheating again.

 

[Joe DeFuria]

HardOCP has updated its news post:

Yeah, not surprsingly, to also note ATI shenanigans...

I think people jumped on the site too quickly; the initial wording did come across in a bad way but it didn't appear to me to be laying blame, rather reinforcing their POV that the benchmark scores may be invalid.

Whatever the case, their POV is just as baseless....

Of course the cheats affect individual test scores as well as the overall score, so it didn't make sense to single that out, and the issue will affect game benchmarking too since the shaders used there will be just as recognisable.

...For that reason.

Some of the cheats being applied to 3DMark can just as well be applied to any game timedemo. So to blame 3DMark for being subject to these issues, when the fact is, 3DMark is willing and able to catch, investigate, and address the issues just goes contrary to that whole POV. This makes 3DMark all the more reliable, and makes you wonder about the validitiy of every other popular benchmark, games included, that have not published such findings.

So instead of [H] saying

* "Hmmm....you know, I wonder what else is going on, particularly in nVidia drivers given the amount of gains they had from 3DMark, and given their exclusivity to Doom3 for example, that might be "artificially" impacting scores...

He's saying

* Hmmm...3DMark admits and addresses cheating occurs in benchmarks. That somehow means further evidence that we shouldn't rely on that benchmark. As for other benchmarks that have not been investigated, to which similar cheats can be applied? Eh...."don't ask and don't tell." Seems that [h] is satisfied for as long as they can plead ignorance.

 

[LeStoffer]

The Quick Verdict:

A to Futuremark for clearing this mess up so relatively quickly and particularly for being bold enough to publish these issues in the danger of upsetting the IHV’s that are the reason for their livelihood.

A+ to ExtremeTech for the courageousness of going public on the issue when the indications was very strong but the verification was still not in.

A to Beyond3d for their support of ExtremeTech’s findings while keeping the issues as unbiased as possible given their status.

F to nVidia. This is absolutely unacceptable. It was a lot worse than anyone would have liked to believe.

C- to ATI. While the jury is still out, any shader optimization (GT4) is bad play. The company has been in the cheat zone before and should have learn their lesson.

D to every website that was to cowardly or afraid to upset the big players to report the issues in a fair manner.

 

[Gunhead]

Maybe, as suggested here, FM needs to start outputting new versions regularly to prevent/catch cheats. But instead of "3.2.0", "3.3.0" etc., they should do it as "3DM03 June refresh", "3DM03 July refresh", an so on, and all the Jeff Ks out there would think this is normal and actually kewl. (Barring Steve, perhaps.)

[As with the current 3.2.0 -> 3.3.0 progression, this wouldn't really hurt benchmarking consistency, because there shouldn't be much difference from refresh to refresh if the drivers behave. A site or magazine would simply use the latest refresh for a crop of cards they review. It would be too painful for cheaters (*cough* *Nvidia* *cough*) to keep up succesfully if it was a monthly alteration.]

I mean, if FM has to do it, why not make the best of it? Make it an outstanding feature instead of a necessary evil

 

[Nite_Hawk]

bjorn:

That's why you should submit any optimizations to futuremark. If they like it, and think that it should go in the benchmark, then it will. If it's only going to help one vendor out, then it might not. Certainly if it helps one, and doesn't hurt the other, then the submitter has a strong case for why it should be inserted in. This of course, is all assuming that the code being inserted is valid DX9 code, and not some backend to a propriatary renderer. Certainly if it's being submitted to put in the code, then atleast all of the beta members will be able to see it and say whether or not it's a valid optimization.

Nite_Hawk

 

[Tim]

If you replace the code such that there is no difference in output, then yes you are still rendering what the programmer intended. However you are not necessarily using the method intended.

I agree, but it really your fault reading what I actually wrote instead of reading what I intending to write. ;-)

It's debatable whether this would be good in games - the better solution is to contact the developer and make sure the original game has a more optimal shader.

I seem to agree with everything that you are writing, It is always better to work together with the developer, but I can think of plenty of cases where replacing shader code could be an advantage to the consumer. Not only if the original code is slow or buggy also if a game uses vendor specific code other manufactures could replace it with code for their cards. I just think that it should always be in the driver notes if they replace shaders.

I only see one problem, if nVidia or Ati is replacing shadercode in games why should game developers put any effort in making optimal shadercode?

However for games that are no longer being patched with updates, the only way to produce this optimisation would be to detect it in the drivers. But this argument doesn't count for a benchmark test - the purpose of that is to see how fast the card is running predefined code. If the code is not the same from card to card no comparison can be made (or rather, a comparison of ability to produce desired effect can be made, but not of speed at specific algorithms). If ATI is replacing the code (which hasn't been proved yet it would seem) with something that produces identical output but is more speed-optimal, it invalidates the tool as a benchmark.

I agree.

 

[Joe DeFuria]

I mean, if FM has to do it, why not make the best of it? Make it an outstanding feature instead of a necessary evil

I think that is absolutely the way to go.

IHVs need to know that there is a MUCH greater risk of being "caught", if they decide to cheat. And if FM some sort of regular "refreshes" to decrease the liklihood of detection schemes working...that makes it very risky to try and optimize based on detection.

The goal is to make detection of cheats high enough in probability, that the IHVs don't risk it in the first place.

 

[Evildeus]

A+ to ExtremeTech for the courageousness of going public on the issue when the indications was very strong but the verification was still not in.

A to Beyond3d for their support of ExtremeTech’s findings while keeping the issues as unbiased as possible given their status.

I'm going to be criticised once more, but well i like it

Why haven't ET/B3D talked of Ati "optimisation" before the PR?

I think it's because of the size of the cheating of NV is not comparative to the "optimization" of ATI. But, shouldn't the same policy be applied whatever is the importance of the issue?

 

[Joe DeFuria]

Why haven't ET/B3D talked of Ati "optimisation" before the PR?

Um, because apparently, there was no way for them to TELL there was a "detection based" optimization going on?

The only reason why B3D and ET talked about the nvdia stuff, was because of the BLATANT RENDERING ERRORS apparent in the developer version of the program.

It's the same reason why they didn't uncover the "replace shader" cheats that nVidia did....because there was no way for them to observe them. They DID run ATI cards in the dev version, and saw no errors.

I think it's because of the size of the cheating of NV is not comparative to the "optimization" of ATI.

No, it's just because of the observableness of the cheat.

 

[Dio]

So, basically, this screams for a benchmark using a HLSL. Then we will be absolutely sure that two different cards won't do exactly the same work , but at least we know that it will correspond to actual gaming performance.

<cough> Dhrystone, Whetstone? <cough>

 

[Joe DeFuria]

And now for a realted question:

Why does Evildues continually try and discredit B3D and ExtremeTech as being anything but impartial observers in all of this? While we don't see the same criticisms from ED about [H]?

 

[Pete]

Two reasons, ED:

1. There were no visibly obvious cheats on ATi cards (no clip planes / lack of buffer clears).
2. No one had FM's 330 patch.

These reasons are so obvious, what am I left to conclude about your motivations or capabilites? You seriously couldn't figure this out yourself? It seems most forum trolls enjoy attention, no matter how it's attracted.

As for ATi's performance boost: How far back does it occur? 3.4, 3.3, 3.2, ... ? Isn't it possible it's just ATi improving their drivers as they and nVidia do for games (surely Quake 3 scores haven't remained the same since the GeForce 3's inception)? As for whether it's appropriate for ATi to improve 3DM performance without altering IQ, I'm not sure where I stand on this. Are IHV's expected to freeze driver development when 3DM is released? Does FM not expect IHV's to optimize (not cheat) for games that may be developed straight to DX9 (budget titles / new developers) if they become popular (like 3DM has)?

FutureMark: Some forum members (of other sites) questioned the validity of your screenshots because they weren't showing the same time or frame. Perhaps it would be wise to include a clarification in your PDF that the reason the numbers aren't the same is because you were in free mode?