New 3DMark03 Patch 330

[RussSchultz]

your actual answer to my question appears how I would answer it: that 3DGuru's decision to drop 3DMark, and only 3DMark is not something that we would agree with. Why don't you just directly answer my question, and say so?

Because that's not what I think. I did not say that their "decision to drop 3dmark, and only 3dmark is not something that I would agree with". That question wasn't asked.

But if that is the question you want answered, and we assume that they are "dropping 3dmark and only 3dmark until things change"(which I don't think is explicitely said--though could be interpreted to):
If we take agree to mean accept: sure, I can accept that as a valid decision.
If we take agree to mean support: sure, I can support that as a valid decision
If we take agree to mean "I would come to the same conclusion and do the same thing": I wouldn't put it outside the realm of possibility. Its tough to really say, since I'm not in the position, and I wouldn't be in that exact position if I were doing benchmarks. If the world was aligned such that I was unable to supplement my test suite with other benchmarks, and was still faced with the choice--I'd probably drop reporting 3dmk03 for now and see how things develop.

 

[AzBat]

I don't know. Since they should be more unsure about this situation, wouldn't it be more understandable if they decided to stop using it for now just for that reason ?

Good point. This gives me an idea/suggestion...

This might be a good case for Futuremark to require all review/tech web sites to become beta members if they're going to publish 3DMark benchmark results. Maybe even go a step further by training and certifying those sites to make sure they understand the benchmark, it's results and how to catch cheating. This way any sites that decide to crap on the 3Dmark tool as useless or not representative of actual games, they won't get to publish the results in reviews. If they do anyway, then at least Futuremark has a legal recourse to go after them. Conceivably sites that weren't beta members might not be considered as being good benchmarkers or ones that don't care if IHVs cheat.

Tommy McClain

EDIT: add/removed words for clarity

 

[Bjorn]

I don't know. Since they should be more unsure about this situation, wouldn't it be more understandable if they decided to stop using it for now just for that reason ?

the question is, if they stop using that, should they not also stop using everything else? We've seen what the IHV's are capable of doing here, and it applies just as much to a game benchmark as 3DMark except that we have people policing 3DMark - who is policing gamebenchmarks?

Afaik, there's no obvious indications of cheating in other benchmarks. At least not yet

 

[Dave Baumann]

Afaik, there's no obvious indications of cheating in other benchmarks. At least not yet

Errr, was it obvious to you that there was something going on in 3DMark until it was pointed out?

 

[Bjorn]

Errr, was it obvious to you that there was something going on in 3DMark until it was pointed out?

No. But now it is. And that's the thing isn't it ?

We now know that Nvidia is cheating in 3D Mark. So, i understand why some sites are f.e waiting for a new pair of drivers from Nvidia (and i guess from Ati also) and some indications from Futuremark if they think that those drivers are from from these type of "cheats" before they continue to use it.

 

[Joe DeFuria]

Because that's not what I think. I did not say that their "decision to drop 3dmark, and only 3dmark is not something that I would agree with". That question wasn't asked.

My question was "do you agree with their decision." Now, to fully clarify that, (which I thought was apparent given the past 2 pages of discussion), "their decision" is actually the sum total of three separate decisions, which is an overall course of actions.

I'll elaborate below.

sure, I can accept that as a valid decision.

I can't. How I cnnot reconcile that decision with the fact that the same cheats can be applied to other benchmarks, difference being those other benchmarks are not policed.

If we take agree to mean support: sure, I can support that as a valid decision

Obviously, I can't. It's inherently inconsistent.

If we take agree to mean "I would come to the same conclusion and do the same thing": I wouldn't put it outside the realm of possibility.

Of course we can't really be sure if we'd come to the same exact conclusion. What I'm getting at is this. We have a "sum total course of action" that they could take. This course of action is comprised of several decisions:

1) Drop 3D Mark, until further notice, or continue to use 3DMark
2) Drop other high-profile benchmarks, or use other high-profile benchmarks
3) "Drop" nvidia until further notice, (or some other form of "punishment") or continue to use nVidia products as if nothing has happened.

You have these decisions to make (among others of course), based on the evidence provided (3DMark PDF and 330 patch), and nVidia's response to that evidence: "probably a bug....FM may be purposely trying to make our products underperform", with no evidence to back that response.

Now, Considering decisions 1,2 and 3 above: Guru3D chose:

1) Drop 3DMark
2) Keep other high-profile banchmarks
3) Continue with nVidia products/drivers as if nothing has happened

That combination of decisions (sum total action) I don't feel is defensible, and of course I don't agree with it. Had ANY ONE of those decisions been made the other way, I could see it as defensible.

I hope my position is clear.

My question(s) to you is
A) Given choices 1,2, and 3 as presented to you, which way would you choose?
B) If it's not the same as Guru3D, why not?

To be clear, I would have probably done the opposite on all three. But again, I can see chaning any ONE of those decisions as something defensible, and merely something I disagree with.

 

[Joe DeFuria]

Errr, was it obvious to you that there was something going on in 3DMark until it was pointed out?

No. But now it is. And that's the thing isn't it ?

No, that's not "just" the thing. The thing is that it was pointed out...and addressed.

We now know that Nvidia is cheating in 3D Mark. So, i understand why some sites are f.e waiting for a new pair of drivers from Nvidia (and i guess from Ati also) and some indications from Futuremark if they think that those drivers are from from these type of "cheats" before they continue to use it.

Again, at least those cheats that were found in the current drivers have been DEFEATED by FutureMark with patch 330. With patch 330, FM HAS "indicated" that you can use patch 330 with the current drivers!

Patch 330 is FutureMarks "indication that even if the drivers have those cheats, 3DMark is free from their effects."

In reality, any NEW drivers coming from nVidia and ATI need to be scrutinized with 3DMark. NOT the current ones!

 

[Bolloxoid]

We now know that Nvidia is cheating in 3D Mark. So, i understand why some sites are f.e waiting for a new pair of drivers from Nvidia (and i guess from Ati also) and some indications from Futuremark if they think that those drivers are from from these type of "cheats" before they continue to use it.

And you also understand that that is exactly what The Big N wants? Sites are playing into Nvidias hands when they are making statements saying they are "waiting" or "unsure" or "undecided" or "in doubt", when the full facts are presented to them. And the facts in this case indicate that the newest build of 3DMark, with reasonable certainty, gets rid of the cheats.

 

[John Reynolds]

And you also understand that that is exactly what The Big N wants? Sites are playing into Nvidias hands when they are making statements saying they are "waiting" or "unsure" or "undecided" or "in doubt", when the full facts are presented to them. And the facts in this case indicate that the newest build of 3DMark, with reasonable certainty, gets rid of the cheats.

Which makes me suspect that two things will happen:

1) NV will never make an official statement on this situation

2) problems with Detonator drivers and 3DMark03 will actually increase with each new release

 

[nelg]

I don't know. Since they should be more unsure about this situation, wouldn't it be more understandable if they decided to stop using it for now just for that reason ?

Good point. This gives me an idea/suggestion...

This might be a good case for Futuremark to require all review/tech web sites to become beta members if they're going to publish 3DMark benchmark results. Maybe even go a step further by training and certifying those sites to make sure they understand the benchmark, it's results and how to catch cheating. This way any sites that decide to crap on the 3Dmark tool as useless or not representative of actual games, they won't get to publish the results in reviews. If they do anyway, then at least Futuremark has a legal recourse to go after them. Conceivably sites that weren't beta members might not be considered as being good benchmarkers or ones that don't care if IHVs cheat.

Tommy McClain

EDIT: add/removed words for clarity

Great Idea ! Perhaps Futuremark could have a certification process for review sites. Not a mandatory one but one that would allow sites to declare that they are CERTIFIED 3DMARK reviewers. The process could ensure that certified members are kept abreast of all the latest developments.

 

[Bjorn]

Again, at least those cheats that were found in the current drivers have been DEFEATED by FutureMark with patch 330. With patch 330, FM HAS "indicated" that you can use patch 330 with the current drivers!

Patch 330 is FutureMarks "indication that even if the drivers have those cheats, 3DMark is free from their effects."

In reality, any NEW drivers coming from nVidia and ATI need to be scrutinized with 3DMark. NOT the current ones!

You're right. But i'm guessing that some of the sites that are unsure wan't to see some continuity in this.

Another thing, are you sure that other benchmarks, f.e games are completely unpoliced afa cheating goes ?

What i mean is, don't you think that Carmack or Sweeney at least does some validating on their engines with regards to benchmarking and different cards/drivers ?

 

[AzBat]

Great Idea ! Perhaps Futuremark could have a certification process for review sites. Not a mandatory one but one that would allow sites to declare that they are CERTIFIED 3DMARK reviewers. The process could ensure that certified members are kept abreast of all the latest developments.

If it wasn't mandatory, then it would basically be what we already have with the 2 beta members(B3D and ET). That hasn't seem to be enough. That's why I think making it mandatory would go along way making sure all sites using 3DMark are on the same level.

Tommy McClain

 

[Joe DeFuria]

You're right. But i'm guessing that some of the sites that are unsure wan't to see some continuity in this.

I understand what you're saying, but my point is, the continuity burden should be with drivers. How can one test continuity without a starting point? That starting point is the current drivers (Cat 3.4, Det 44.03), and 3DMark with 330 patch.

The sites should be using 3DMark, and watching to see if there are any significant performance increases with newer drivers. If there are, THEN you look at the drivers, and possibly a response from 3DMark, to see if a perormance boost is due to a cheat or not.

Another thing, are you sure that other benchmarks, f.e games are completely unpoliced afa cheating goes ?

I think as long as the game renders reasonably correctly and with decent performance, the games not not "policed" beyond that.

ID did not find Quak, for example. I certainly don't think they'll be checking every new driver set to see if there are clipping planes being inserted for timedemo benchmarks. That's not their job.

 

[Joe DeFuria]

Which makes me suspect that two things will happen:

1) NV will never make an official statement on this situation

2) problems with Detonator drivers and 3DMark03 will actually increase with each new release

Agreed. I see a battle of "who can do what faster: nVidia introducing drivers with more cheats (more stealthy detection mechanisms) to pump up the score, or patch 3DMark to defeat them."

i.e:

Det 50.0: 3DMark up 1000 points:
Path 331: 3Dmark down 1000 points.
Det 50.1 3DMark up 1000 points
Patch 332: 3DMark down 1000 points

Ad nauseum.

And the sad thing is, if the reaction to this patch 330 is any indication, web sites will "interpret" that as "how can we trust 3DMark"?

 

[Bjorn]

I think as long as the game renders reasonably correctly and with decent performance, the games not not "policed" beyond that.

ID did not find Quak, for example. I certainly don't think they'll be checking every new driver set to see if there are clipping planes being inserted for timedemo benchmarks. That's not their job.

Well, they shouldn't have to check for "inserted clipping planes" since you can record your own timedemos and also actually play the game. Something you can't do with 3D Mark unless you become a beta member. Well, maybe not play but at least roam around freely. I think someone mentioned that they should put these things in the Pro version which seems like a good idea to me.

And i agree with you, they're probably not doing any extensive checks with every new driver that are released from each vendor.
But i wouldn't go as far as saying that they're not doing anything with regards to these things.

 

[nelg]

If it wasn't mandatory, then it would basically be what we already have with the 2 beta members(B3D and ET). That hasn't seem to be enough. That's why I think making it mandatory would go along way making sure all sites using 3DMark are on the same level.

Tommy McClain

I just feel that if it was mandatory it might discourage some sites from using it. That would not be in Futuremarks interest. I am not suggesting a level of involvment like that of a beta member. Just a way to keep sites using the benchmark correctly. If a review of an ATI or Nvidia card was done today with the unpatched 3Dmark03 look at how it may protray these cards. If there was a certification process maybe some sites would be cautious about doing such things for if they did they would loose there certification. As a layman in this area I personnally would feel better if a site was labeled "CERTIFIED by FUTUREMARK".

 

[LeStoffer]

And you also understand that that is exactly what The Big N wants? Sites are playing into Nvidias hands when they are making statements saying they are "waiting" or "unsure" or "undecided" or "in doubt", when the full facts are presented to them. And the facts in this case indicate that the newest build of 3DMark, with reasonable certainty, gets rid of the cheats.

Which makes me suspect that two things will happen:

1) NV will never make an official statement on this situation

2) problems with Detonator drivers and 3DMark03 will actually increase with each new release

Which make me fear that have nVidia reached their short term goal, namely to inflict some lethal damage to 3Dmark03. As long as the market leader dismissed the benchmark, a big part of the industry will - for a good reason - a wary of using it.

Don't make the mistake to think that nVidia didn't take all the consequences of their decisions into account: In the short term it was a win-almost win situation: If they didn't get caught, brilliant. When they did, well, at least 3Dmark03 was rendered obsolete at the same time.

I applaud Futuremark for trying to be in the game with the patch, but nVidia is far to strong in this business for 3Dmark03 to regain benchmark status without nVidia's backing. You may not like this - I know I don't - but there's not much we can do about (beyond calling OEMs like MSI and Dell to take a stance).

In the long term, however, I do think that nVidia overestimated how much their market leader position would allow them to decide on behalf on the rest of us/the industry. I know I lost respect for the company after this, which is really frustrating given the huge mass of world class talent and nice, innovative stand-up guys that works there.

I cant help feeling that something is rotten in the state of nVidia management...

 

[Nick[FM]]

Maybe even go a step further by training and certifying those sites to make sure they understand the benchmark, it's results

Actually.. I have been planning / throwing some ideas around (in my head only so far) to organize a "Benchmarking Bootcamp - Online" or whatever you want to call it. It could either be once a month, an ongoing thing, or then once a year (bigger happening). At least something where online, and offline, media could come to discuss about 3DMark & PCMark, listen to "lectures" etc. about our products. To get the best possible guidelines how our products are to be used / not be used, and maybe some instant help if there are some issues / problems.

"Certified 3DMark Site" or something similar would be a good thingâ„¢. The idea is nice! Maybe some test in order to get it? Hehe..

Question1: How many Game Tests are there in A) 3DMark99 B) 3DMark2000 C) 3DMark2001 and D) 3DMark03
Question2: Using what formula is the CPU score calculated in 3DMark03?
Question3: What did the trolls in GT3 (3DMark03) do when the heroine interrupted them?
Question4: How many separate speeders were there in 3DMark99 GT1?
Question5: Who made the music for A) 3DMark99 B) 3DMark03

Hehe..

But seriously, I will post more about this if it will ever become a reality. Any ideas & suggestions concerning it, please email me! It's better that way as I might miss some good pointers if they are posted in some thread somewhere. Of course posting here is ok too, but there's always a chance that it gets unnoticed.

 

[RussSchultz]

3) Continue with nVidia products/drivers as if nothing has happened

And that, I believe, is where we differ. I don't think they are continuing "as if nothing has happened". I think their blog was pretty clear that they expect the cheating to continue (and that to me means that it won't be business as usual)

But far be it for me to tell you what to think, or how to interpret other people's opinions. You asked me for mine; I gave it to you.

Now accept it; because you aren't going to change it.

We've filled up another page of words. Isn't that nice?

 

[Joe DeFuria]

We've filled up another page of words. Isn't that nice?

Not nearly as nice as actually answering my direct question. See, by answering my question, you actually do more than just fill up the page with words. You give me insight to your actual position.

I'll restate the quesiton.

My question(s) to you is
A) Given choices 1,2, and 3 as presented to you, which way would you choose?
B) If it's not the same as Guru3D, why not?

To be clear, each of the three questions below has two options. Which option would you choose with each of the three questions?

And questions 1,2, and 3 are:

1) Drop 3D Mark, until further notice, or continue to use 3DMark
2) Drop other high-profile benchmarks, or use other high-profile benchmarks
3) "Drop" nvidia until further notice, (or some other form of "punishment") or continue to use nVidia products as if nothing has happened.

(Yes, I understand that with respect to question 3, you interpret 3DGuru as not going to treat nVidia products with "business as usual" attitude.)