Futuremark has problems here.

[Joe DeFuria]

]So what you are saying is that downloading a new patch for 3DMark (& installing it) everytime a new driver (from any IHV) is released would be less hassle than simply installing the drivers we have approved?

Yes, but it's more than that.

It's very logical and "common sensical" for reviewers to run tests with all the "latest" updates as possible.

That is:
1) Get the latest drivers
2) Get the latest application patches.

Be it 3DMark, or Quake, or any other app / benchmark, reviewers typically check for the "latest versions" before running their set of benchmarks.

It is NOT simply a matter of installing the drivers that you approve, because 3DMark is certainly not going to be the only benchmark they run. So you're asking the reviewers to run one set of benchmarks with the IHV recommended drivers, and your "special" benchmark with an older driver.

What happens?

1) Reviewers think this is a PITA
2) Reviewers question "what is wrong with 3D Mark?" (Even though the right question is "what is wrong with nVidia".)
3) Reviewers just genuninely get confused about the whole thing...and the end result is "well, if I don't understand what's going on, I'm not going to use it."

Case in point, LegitReviews (as in the other thread

So as of today, January 26, 2004, we will no longer include 3DMark03 in our graphics solution reviews in order to use the most recent graphics driver in all of our tests.

Now, You and I know that the "point" of benchmarks is not "to use the most recent graphic driver." The point of benchmarks is to be "fair" and "representative" of the hardware to the greatest extent possible. We know that legit reviews has it backwards.

That doesn't change the fact that they jumped ship. In all liklihood, they just don't care to take the time to understand 3DMark's position in this, so they say "ah, screw it."

Now, you provide a patch so that they CAN use 3DMark with "the latest drivers", and they'll have no problems. just as they have no problem downloading and installing the latest patches for games.

 

[andypski]

Now, you provide a patch so that they CAN use 3DMark with "the latest drivers", and they'll have no problems. just as they have no problem downloading and installing the latest patches for games.

This is a losing proposition Joe -

Every time Futuremark relases a patch they 'show their hand' as it were - in other words as soon as the patch goes out then interested parties have access to it and can begin circumventing whatever measures were put in place to disable the application detection. As long as Futuremark keeps their code to themselves and simply tests each driver the IHVs do not have this ability, or at least it is far more difficult.

You have to realise that there can only be a finite number of ways in which Futuremark can work around application or shader detection before they reach the point of altering the benchmark sufficiently that the scores will start to change as a natural consequence. If they keep revealing their hand I would expect that the point where the manipulations are too great might actually be reached quite quickly, at which point they start to have problems with people feeling that the benchmark has been invalidated because no previous scores are comparable with current ones.

 

[Mark]

maybe an anti-detect mode, similar to what ShaderMark uses.

I'm sure if it was doable then they would have already done so by now though... maybe for 3DMark04

 

[andypski]

maybe an anti-detect mode, similar to what ShaderMark uses.

And what exactly do you think this mode does that is significantly different to what Futuremark do to get around driver detection of 3DMark?

I don't believe that there is anything particularly magical about the anti-detect mode in Shadermark that prevents IHVs from working around it should they choose to do so.

 

[Quitch]

]

Yes. A patch doesn't require a system rebuild (as changing drivers would to maintain fair results). A patch number makes for easy user comparison, if they see an old version number then they will complain.

But but.. Why would anyone complain about a benchmark build number, but not about a driver version is beyond me.. I see your point, but I am not sure if it would be any less work for the reviewers & users. I think that only a very small minority makes a clean install of Windows everytime they change drivers, and not sure how many reviewers do it. Uninstalling drivers and installing new ones work pretty perfectly these days. A professional reviewer would notice if there was something wrong with the driver update/downgrade when looking at the first benchmark numbers. Dunno. But releasing a patch & full install for 3DMark everytime a new driver from any IHV is being released is .. well, not directly impossible, but next to impossible. Sometimes there can be driver releases from different IHV's only 7 days apart, which would mean 2 patches / week. Not a really reasonable option IMO.

It is standard practice to rebuild the test machine for every card, how else could you ensure fair results? Because of this, it is, while not a lot of work, time consuming to swap drivers. On the other hand, reviewers already patch 3DMark03, so using patch 350 instead of 340 is no extra work at all.

I don't suggest for a second that most people rebuild their systems for this, but most reviewers DO, and it's for reviewers that these patches are needed, because then you get pressure from readers to use the latest version, but there's no pressure to use old drivers, something the mindset of the user considers silly.

A user is always looking for the latest number, and if it isn't there, they'll protest. Guidelines take reading, but version numbers, well, you just need to know it's the latest, something that takes no effort at all. That's how users work.

 

[Quitch]

What a lot of people seem to miss is that reviewers are driven by their readers, and most sites are outside of the realm of specialising in graphics. A user will look at the version number, and they will likely know if it was the latest one at the time, it is EXPECTED of a reviewer to use the latest patches and drivers. This isn't because it's right, simply because it's the way things are, it's a mindset that the latest patch/driver is always best.

Users won't know about guidelines, they can't hold reviewers to them. Try to explain them, and debate rages in the forum, and many users will simply blame Futuremark. Patches are a recognised and accepted standard, guidelines which prohibit drivers are not. Just look at what happened on Adrian's Rojak Pot.

I'm not saying patches are the way you should want to go, but they are a way that will work. Guidelines will never work, unless there is a major shift in the user mindset. Considering only Futuremark are going around, enforcing this sort of thing, I don't see that happening.

 

[andypski]

I'm not saying patches are the way you should want to go, but they are a way that will work.

They will not work for long, for the reasons I gave above.

 

[Quitch]

Longer than the driver guidelines will, as has been seen. I doubt nVidia will invest much time in 3DMark03 once 04 is around, as any sites intending to stick with FutureMark products will likely make the switch, so the fix is only for so long.

 

[andypski]

Longer than the driver guidelines will, as has been seen. I doubt nVidia will invest much time in 3DMark03 once 04 is around, as any sites intending to stick with FutureMark products will likely make the switch, so the fix is only for so long.

And why would Futuremark not then just run into exactly the same problems with 3DMark04 that they have had with 03?

 

[Bouncing Zabaglione Bros.]

Longer than the driver guidelines will, as has been seen. I doubt nVidia will invest much time in 3DMark03 once 04 is around, as any sites intending to stick with FutureMark products will likely make the switch, so the fix is only for so long.

And why would Futuremark not then just run into exactly the same problems with 3DMark04 that they have had with 03?

I guess this is why so many people want Futuremark to publicly shame Nvidia into stopping these cheats by (1) publishing info on drivers that fail to be approved due to cheats, and (2) ban any uploads to ORB using cheating drivers.

The only solution to this problem is to stop Nvidia from cheating. This will not happen while Futuremark refuse to sanction Nvidia (one of their members) from deliberately cheating the results.

 

[digitalwanderer]

I guess this is why so many people want Futuremark to publicly shame Nvidia into stopping these cheats by (1) publishing info on drivers that fail to be approved due to cheats, and (2) ban any uploads to ORB using cheating drivers.

The only solution to this problem is to stop Nvidia from cheating. This will not happen while Futuremark refuse to sanction Nvidia (one of their members) from deliberately cheating the results.

Damn, I never thought of it quite that way but you're really on to something with that....it just feels right to me.

Mebbe that's why I'm so out for nVidia's blood, I really don't want to see them get away with everything they've been getting away with or else it'll just become the status quo rather than shockingly bad behavoir. :|

 

[Bouncing Zabaglione Bros.]

I guess this is why so many people want Futuremark to publicly shame Nvidia into stopping these cheats by (1) publishing info on drivers that fail to be approved due to cheats, and (2) ban any uploads to ORB using cheating drivers.

The only solution to this problem is to stop Nvidia from cheating. This will not happen while Futuremark refuse to sanction Nvidia (one of their members) from deliberately cheating the results.

Damn, I never thought of it quite that way but you're really on to something with that....it just feels right to me.

Mebbe that's why I'm so out for nVidia's blood, I really don't want to see them get away with everything they've been getting away with or else it'll just become the status quo rather than shockingly bad behavoir. :|

If I was one of the other FM partners (like for instance ATI), I'd be putting pressure on FM to be doing the above sanctioning of Nvidia. Why should the other FM members stand for Nvidia cheating? Why should they associate themselves with that way of doing business?

Nvidia is clearly breaking the letter and spirit of the FM guidelines, but they continue to do so because there has been no sanction from FM. Nvidia need to be publicly shamed and if they continue to cheat, kicked out from the 3DMark programme.

This is what should be done if FM were serious about policing (and being seen to be policing) 3Dmark. Instead we have a half hearted effort to control cheating that fails to work, but allows FM to continue to take money from Nvidia, claim they have broad industry support, and allows Nvidia to continue to cheat.

 

[Mark]

Futuremark doesn't have the balls to stop nVidia, we've seen that already last August when they retracted their first report on how nVIdia was cheating.

The only thing they will do will continue the path they are already on by releasing 3DM03 patches that defeat the cheats. Pretty stupid path if you ask me as nVidia will only quickly release another cheat driver.

 

[digitalwanderer]

Futuremark doesn't have the balls to stop nVidia, we've seen that already last August when they retracted their first report on how nVIdia was cheating.

Yeah, but I sort of thought they realized that they came damn close to killing themselves the first time by totally caving to nVidia and were making a stand this time to keep their benchmark's integrity.

I guess it was just another case of Futuremark is trying to put the best possible face on to their current caving in to nVidia's demands.

(And about now is time for Worm to come here and say that FM is doing a whole bunch to make sure no one cheats...a whole bunch alright, a whole bunch of nothing. )

 

[Tim Murray]

the approved driver list would be fine if it were REALLY BLATANTLY OBVIOUS what the approved drivers were. put it on the download page (hell, put it on the front page!), and this problem will magically disappear.

oh, yes, when a result is submitted to the ORB using a non-approved driver, a Big Message Should Pop Up. that'd be enough; Futuremark is doing a terrible job of getting the proverbial word out. the actions they're taking are fine; the steps they're taking to promote said actions are not.

 

[digitalwanderer]

Futuremark is doing a terrible job of getting the proverbial word out. the actions they're taking are fine; the steps they're taking to promote said actions are not.

Kind of like saying, "They're talking the talk, but not walking the walk".

 

[Tim Murray]

no, it's more like one hand does not know enough of what the other hand is doing.

 

[karlotta]

"Bouncing Zabaglione Bros."



"If I was one of the other FM partners (like for instance ATI), I'd be putting pressure on FM to be doing the above sanctioning of Nvidia. Why should the other FM members stand for Nvidia cheating? Why should they associate themselves with that way of doing business?"

.... maybe because the some of the other driver makers cheat too....

 

[Tim Murray]

.... maybe because the some of the other driver makers cheat too....

if ATI does it, nobody's been able to prove it. plus, if they do it in 3DMark, their default 3DMark performance must be really low, since the delta in 3DMark scores is pretty representative of the delta in most benchmarks (especially PS2.0).

XGI, on the other hand--wowiezowie.

 

[Mark]

I haven't seen any evidence of ATI cheating in 3DMark03. I actually talked to a couple guys who said that they wanted to reintroduce their shader optimizations for it after FM said they were legit but the "higher-ups" would rather their hardware stand on its own merits and not get caught up in the whole mess. I don't know if those same shader optimizations made it back in in recent Catalyst releases, has anyone checked?