New 3DMark03 Patch 330

[Joe DeFuria]

I am not sure about this approach. 3dmark is made so that it approximates the game performance.

Well, right off the bat, you are wrong.

3DMark is NOT made to approximate game performance, it is made to stress the hardware in a consisent way. There is a significant difference there. And this is the reason why 3DMark is a valuable tool in addition to actual game performance.

We all know that games normally have specific codepaths for specific hardware. Each GPU ( nv/ati ) does certain things a lot better for others. So normally a game dev(with guidance from an IHV) would optimise for these things to get the maximum performance possible.

Disagree.

SOME will, some won't. And you'll have look at a game by game basis to see which developers use mulitple code paths and which ones don't.

Now 3dmark does most of these things in a fixed way (or the dx9 way) and doesnt have a specific codepath. So this in effect makes its performance not correspond with that of what that particular gpu is fully capable off.

NO single benchmark can test "what the GPU is fully capable of". No game, no synthetic test. All that can be done is to test "How capable each GPU is at running this code, using this data set." 3DMark strives for consistency in code and data set.

Should futuremark also use the approach of game devs and have specific codepaths to remove all doubt from everyones mind ?

No, because we can already use games to answer those types of questions. (At least, for the specific games that are benchmarked.) I think FM should stick with their current policy: keep as 100% consistent as possible.

 

[Nite_Hawk]

Anode:

Well, that's a problem. How many games recieve the optimizations? If you just want to see how well a specific game runs on a card, you test the game (including all of it's driver optimzations). 3DMark (imho) should be a test that tells you how good a card is at specific things before hand tuning takes place. It should be an indication of how well that card can do things in a standard way.

Beyond this, a lot of the optimizations that Nvidia did in 3dmark couldn't be done during game play, only in time demos. Even if you consider it an optimization, it's a misleading one, as it won't help you when you play games. The shaders as another example, aren't the same ones the developer intended you to see. Maybe they are more efficient, but that should be the developers job to change as they see fit, not the drivers.

Nite_Hawk

 

[Doomtrooper]

I am not sure about this approach. 3dmark is made so that it approximates the game performance. We all know that games normally have specific codepaths for specific hardware. Each GPU ( nv/ati ) does certain things a lot better for others. So normally a game dev(with guidance from an IHV) would optimise for these things to get the maximum performance possible.
Now 3dmark does most of these things in a fixed way (or the dx9 way) and doesnt have a specific codepath. So this in effect makes its performance not correspond with that of what that particular gpu is fully capable off. So hence you might see optimisations from IHV's to show truly what their gpu is capable off in 3dmark. They just cant say "to hell with 3dmark" since a lot of people and OEMs use it to judge a gfx product.
The question here is what is the line between such optimisations and cheating. Should futuremark also use the approach of game devs and have specific codepaths to remove all doubt from everyones mind ?

Only in OpenGl would these 'codepaths' be allowed, in Direct3D there is no other code paths, it is all standard.
So if we look at the water shader, it has probably been ran through with CG using their HLSL, more than likely using Partial Precision which is 'supposed' to AFAIK be not allowed in WHQL drivers.

This is the problem when we have multiple HLSL, when really all the effort should go into making the 'two major' HLSL better.

Pocketmoons benchmarks show the Partial Precision speed:

http://www.pocketmoon.com/Cg/Cg.html

 

[Evildeus]

optimisations/cheats...... what's that look like to you, ED? Looks like the word "cheats" to me..... And, at this point, can you admit that nVida is cheating?......

/= or for me so, it means what i mean
Yes Nv is cheating according to FM

 

[Moose]

HardOCP's reply is on their front page. They blame futuremark.

Nite_Hawk

...removes Hardocp from list of favorites.....

 

[WaltC]

front page]
3DMark Patch:
Futuremark has released a patch for 3DMark 2003 that eliminates “artificially high scoresâ€￾ for people using NVIDIA Detonator FX drivers. This is in response to the news item we posted last week ( as did several sites ). To us, things like this just solidify our belief that 3DMark03 overall score is useless as a real world benchmark. Thanks to everyone who sent this one in.

This is really sad....

You know, apparently this issue has simply flown right over the heads of the people staffing [H] to the degree that even when it is explained in an itemized fashion, they cannot understand what has occurred. Their position is that the fact that nVidia cheated 3DMark means that the benchmark itself is "useless," and the fact that since the patch forces the Detonators to render normally without cheating and lowers the aggregate score to that well below a similarly clocked R9800P means absolutely nothing.

Wow. Talk about gluttons for punishment! *chuckle* Who in his right mind would want to buy a product that the manufacturer falsely advertises as having a level of performance much higher than it actually has??? Whew! Not me.... These guys have really put on the blinders...this is taking CYA to an entirely new level.

 

[mczak]

Does this new patch also change the results of older cards, such as GeForce 4Ti or Radeon 8500/9000? At least the back buffer clear optimization sounds to me like it could be done on any card, but OTOH I can't remember seeing much performance improvements from new drivers on these cards.

Oh and btw can we have a anti-cheat 3mark2001se too? I wouldn't be surprised if there are some more cheats to detect there as well... (though you probably can't replace shaders there, as they don't have much influence on the score).

 

[Brent]

front page]
3DMark Patch:
Futuremark has released a patch for 3DMark 2003 that eliminates “artificially high scoresâ€￾ for people using NVIDIA Detonator FX drivers. This is in response to the news item we posted last week ( as did several sites ). To us, things like this just solidify our belief that 3DMark03 overall score is useless as a real world benchmark. Thanks to everyone who sent this one in.

This is really sad....

You know, apparently this issue has simply flown right over the heads of the people staffing [H] to the degree that even when it is explained in an itemized fashion, they cannot understand what has occurred. Their position is that the fact that nVidia cheated 3DMark means that the benchmark itself is "useless," and the fact that since the patch forces the Detonators to render normally without cheating and lowers the aggregate score to that well below a similarly clocked R9800P means absolutely nothing.

Wow. Talk about gluttons for punishment! *chuckle* Who in his right mind would want to buy a product that the manufacturer falsely advertises as having a level of performance much lower than it actually has??? Whew! Not me.... These guys have really put on the blinders...this is taking CYA to an entirely new level.

It hasn't flown over MY head. I understand it.

 

[Doomtrooper]

3DMark Patch:
Futuremark has released a patch for 3DMark 2003 that eliminates “artificially high scoresâ€￾ for people using NVIDIA Detonator FX drivers. This is in response to the news item we posted last week ( as did several sites ). To us, things like this just solidify our belief that 3DMark03 overall score is useless as a real world benchmark. Thanks to everyone who sent this one in.

Well to me that is the exact opposite, here we finally have a benchmark that is policed, and kept at a level playing field. Unlike 3Dmark 2001 which LOD hacks, and cheats were easy to do.

I think most people agree that their opinion is exactly the opposite of [H].

 

[Tim]

And what evidence leads you to assume that? I'm not saying I know something but I think it's way, way too early to make such assumptions.

There are no image quality degradation with Ati´s drivers and they have no problems with the free lock mode. Other than the performance improvement there seems to be no difference, this indicates that it is optimizations not cheats. (Ati could off cause be using some kind of free mode detection).

 

[Nazgul]

It hasn't flown over MY head. I understand it.

Well then, would you mind beating it into Kyle's head with a large blunt object a few times?

 

[WaltC]

It hasn't flown over MY head. I understand it.

You know what? I had intended to add "With the possible exception of Brent..." to my comments. Thanks for reminding me and confirming it for me...

 

[Deflection]

front page]
3DMark Patch:
Futuremark has released a patch for 3DMark 2003 that eliminates “artificially high scoresâ€￾ for people using NVIDIA Detonator FX drivers. This is in response to the news item we posted last week ( as did several sites ). To us, things like this just solidify our belief that 3DMark03 overall score is useless as a real world benchmark. Thanks to everyone who sent this one in.

This is really sad....

You know, apparently this issue has simply flown right over the heads of the people staffing [H] to the degree that even when it is explained in an itemized fashion, they cannot understand what has occurred. Their position is that the fact that nVidia cheated 3DMark means that the benchmark itself is "useless"...

They don't in fact say that. They say the overall score is useless. This has been a distinction for them in the past.

 

[MuFu]

It hasn't flown over MY head. I understand it.

Yes - that's quite obvious from your recent posts here.

Hey, when you're banned from an [H] e-mail inbox does that filter you from all "@hardocp.com" addresses?

MuFu.

 

[ZoinKs!]

It hasn't flown over MY head. I understand it.

Well then I hope you get a chance to put it into terms that Mr. Bennet and the readers of HardOCP will understand. A lot of people out there don't get it.

 

[Dave Baumann]

Yes, we're a beta member and we still maintain the score isn't what should be focused on (although I can see why its there).

 

[Nite_Hawk]

Brent:

I've liked your reviews, and I respect your opinion. So far I've seen nothing that you've written which I'd say is false or misinformed. Having said that, I can't understand how Steve and Kyle could continue to ignore what nvidia is doing and attack futuremark instead. These things could just as easily be done in a time demo for any videogame out there, and yet it's not even mentioned. I'm sorry to say it, but the respect and reputation that *your* articles brought to HardOCP are lost by their words. Brent, I will certainly continue to read what *you* write, but HardOCP as a whole is a tarnished name.

Nite_Hawk

 

[jjayb]

removes Hardocp from list of favorites

I just went over and looked at Steve's response. It is removed from my favorites as well. I can honestly say that was the last time my computer will ever load Hardocp. I am personally boycotting Hardocp. To try to belittle what is going on by blaming 3dmark is ridiculous. At least with futuremark you can be sure it is policed to detect cheating. Hardocp's last 2 statements on this issue are disgusting.

 

[WaltC]

They don't in fact say that. They say the overall score is useless. This has been a distinction for them in the past.

Which has zero to do with nVidia cheating the benchmark as documented. Their problem is that whatever their position might be on the "overall score" that is no defense for what nVidia has done. The two are entirely separate issues--the cheating is not open to debate; the efficacy of the overall score might be. Two very different issues not related to each other. That's the mistake they've consistently made at [H] (with the exception of Brent, who understands the issues.)

One thing which cannot be argued, however, is that nVidia certainly sees an efficacy in the overall score, else it would not have cheated to raise it.

 

[Mummy]

And what evidence leads you to assume that? I'm not saying I know something but I think it's way, way too early to make such assumptions.

There are no image quality degradation with Ati´s drivers and they have no problems with the free lock mode. Other than the performance improvement there seems to be no difference, this indicates that it is optimizations not cheats. (Ati could off cause be using some kind of free mode detection).

Tim your position is bull...

Even with NV drivers there is no degradation, unless you change the Camera's "rail" path, i dont know if ATi is cheating (more for the "they are" than the "they arent" though, just my impression), but *ANYTHING* made to speedup a *SYNTHETIC BENCHMARK* is a *CHEAT*, doesnt matter if every pixel is the same.

HardOCP(Kyle and Steve mostly, not Brent) position on the whole issue is truly ridicle, im really tempted to remove the site from my favorites list, even if they have always many news