Futuremark has problems here.

[Nick[FM]]

We will maintain a list of drivers we have determined to be in accordance with the guidelines. We will
publish detailed description of this process including actions when violations have been discovered.

Uhm, Worm? You didn't really answer the man's question, you either misunderstood or side-stepped it.

He wanted to know what specifically you didn't approve of in the 53.03 set I believe, as well as what actions you plan to take. (See? Your own statement actually implies that you're going to take more action than just not posting the drivers to the approved list. )

I didn't misunderstand him (I think/hope anyway). I just corrected him as he used a quote from our Q&A as basis on something we have never said. It seems that some have the impression that the "We will publish detailed description of this process...." meant that we will publish detailed info on drivers (why they didn't pass etc), whereas it meant that we will publish the Process to Enforce the Optimization Guidelines document.

If you check out the pdf, you should be able to see what we do (ie. "actions") if a driver is not approved.. and besides, I think I have explained it a couple of times already.

A driver is not approved if it doesn't fulfill the run rules. The run rules can be found in Process to Enforce document.

 

[Nick[FM]]

Is it technically too difficult/impossible for Futuremark to implement a sort of anti-detect mode, like in ShaderMark2?

It would be possible, but there is no 100% bulletproof anti-detect system, so it wouldn't really be worth the trouble.

 

[Deathlike2]

We will maintain a list of drivers we have determined to be in accordance with the guidelines. We will
publish detailed description of this process including actions when violations have been discovered.

I think what is meant by "publish detail description" is that besides the offender being informed... we the PUBLIC would be informed AS WELL.

If you check out the pdf, you should be able to see what we do (ie. "actions") if a driver is not approved.. and besides, I think I have explained it a couple of times already.

I doubt those reviewing 3DMark03 have actually read this document either.

In fact, with respect to any Futuremark document, I doubt many major reviewers have read it. I'd expect that most of them "benchmark and go" meaning that they'll benchmark it and not think about it that much more (like say getting the lastest 3DMark patch and/or checking that approved driver list, let alone knowing of the existance of either).

 

[AzBat]

]

We will maintain a list of drivers we have determined to be in accordance with the guidelines. We will
publish detailed description of this process including actions when violations have been discovered.

Uhm, Worm? You didn't really answer the man's question, you either misunderstood or side-stepped it.

He wanted to know what specifically you didn't approve of in the 53.03 set I believe, as well as what actions you plan to take. (See? Your own statement actually implies that you're going to take more action than just not posting the drivers to the approved list. )

I didn't misunderstand him (I think/hope anyway). I just corrected him as he used a quote from our Q&A as basis on something we have never said. It seems that some have the impression that the "We will publish detailed description of this process...." meant that we will publish detailed info on drivers (why they didn't pass etc), whereas it meant that we will publish the Process to Enforce the Optimization Guidelines document.

If you check out the pdf, you should be able to see what we do (ie. "actions") if a driver is not approved.. and besides, I think I have explained it a couple of times already.

A driver is not approved if it doesn't fulfill the run rules. The run rules can be found in Process to Enforce document.

Worm, Dig, etc

For the past couple of weeks I've been suspecting that this what FutureMark considered to be the response to the quote you guys have mentioned. And if you think about it I believe you'll agree the Process to Enforce PDF does comply with that previous quote. So if you take it from there what does the Process to Enforce PDF state what they're going to do when they find drivers that don't comply with the guidelines? Here it is...

There are two possible outcomes of the above:

1) Futuremark’s suspicion was proved to be wrong and in fact the driver met the Optimization
Guidelines. In this case, Futuremark informs its BDP members of the results and publishes
information of the driver in its web pages with status: ‘Reviewed’; or

2) The driver did not fulfill the Optimization Guidelines. In this case, Futuremark informs its BDP
members of the results and does not publish information of the driver on its web site

If reasonable evidence is provided, a driver can still be reviewed again at a later time. Futuremark gathers
feedback from users of its products, and will make further inspections if needed.

Looks like they're doing exactly what they said they were going to do. Do I agree with it? Not really, but I'll talk about it more in another message when I get done reading Joe's response.

Tommy McClain

 

[Nick[FM]]

It's the obligation of the reviewers to download the latest 3DMark patch, just as they are "obligated" to download the latest drivers from the IHVs.

So what you are saying is that downloading a new patch for 3DMark (& installing it) everytime a new driver (from any IHV) is released would be less hassle than simply installing the drivers we have approved? I am not sure if I see the logic in that. :? Or then my head simply isn't working well (terrible flu.. ).

 

[Quitch]

Yes. A patch doesn't require a system rebuild (as changing drivers would to maintain fair results). A patch number makes for easy user comparison, if they see an old version number then they will complain.

 

[demalion]

Where I depart from Joe most distinctly is the idea that producing continual patches is feasible for Futuremark.

The process, AFAICS:

• costs time and money (as it does nVidia to continue to cheat, but nVidia has more resources)
• is of limited benefit (due to nVidia persisting in defeating it again in the next driver version that they will release anyways)
• provides a perpetual strawman for nVidia to attack in terms of blame (not a logical one, but an effective one to those uninformed about the technical issues)

This is a losing proposition in practical terms, to the best of my knowledge.

Where I agree is in the idea that Futuremark is failing to take certain steps that do seem to be within their means and are not losing propositions:

• Failing to deal with the PS 2.0 test was a significant mistake, because it throws away a test (this doesn't mean it wasn't an oversight, or even understandable, it just means it is a significant point of failure that weakens credibility).
• There are steps Futuremark could indeed take and information they could make more accessible that would markedly improve the success of what they did do...not because it is their fault that these steps are necessary, or their fault that some reviewers are irresponsible, but because their goal should be to succeed without concern of blame. To clarify, it is akin to advertising...for a good company trying to make a good product, it isn't just a matter of making the good product and suffering when inferior products sell better because of more advertising, it is a matter of taking all the available opportunities that are within your means to illustrate how your product is a good product. The ORB information ideas that have persistently been presented, and even the "watermark" idea as recently repeatedly proposed by dw, are indeed silly when evaluated by the criteria of the information already being provided, but not at all silly, and even essential, from the perspective of opportunities Futuermark has available to them to get the message out, and are passing up.

The continued problem, AFAICS, is that Futuremark is caught up in the blame game nVidia has tried to define this situation as, and have lost sight of the larger perspective of their relationship to their users outside of the situation nVidia is working to define. Their responses seem to have been primarily geared at reactively dealing with things within the framework of blame in which they were attacked, and not taking the best advantage of the other avenues available to them outside of that framework.

 

[Nick[FM]]

Yes. A patch doesn't require a system rebuild (as changing drivers would to maintain fair results). A patch number makes for easy user comparison, if they see an old version number then they will complain.

But but.. Why would anyone complain about a benchmark build number, but not about a driver version is beyond me.. I see your point, but I am not sure if it would be any less work for the reviewers & users. I think that only a very small minority makes a clean install of Windows everytime they change drivers, and not sure how many reviewers do it. Uninstalling drivers and installing new ones work pretty perfectly these days. A professional reviewer would notice if there was something wrong with the driver update/downgrade when looking at the first benchmark numbers. Dunno. But releasing a patch & full install for 3DMark everytime a new driver from any IHV is being released is .. well, not directly impossible, but next to impossible. Sometimes there can be driver releases from different IHV's only 7 days apart, which would mean 2 patches / week. Not a really reasonable option IMO.

 

[cthellis42]

Forcing continual patching wouldn't be thrilling, but if 3DMark were built with an auto-updater, THAT would allow Futuremark to stay ahead of anyone's optimizations easily and keep its users informed right within the program itself. Yes, it would require they be online, but so would downloading patches. I don't think that's a HUGE issue... (And at the very least, should they choose to run without updating, there could be reminders placed right in front of them, so people wouldn't have to get headaches from "oh so much confusion! <sob>") Running the program itself should be the way for the public--Joe Q. and reviewers alike--to stay informed.

 

[Deathlike2]

I think that what the problem is that even though Futuremark is actively doing what you set out to do is ok, but the majority of users (reviews and normal users) don't see that being done.

The "hassle" of those reviewers to check Futuremark's approved drivers page does seem to be a real hassle. It is more of an reaction to something Futuremark hasn't stressed:

1) Reviewers don't review the agreement

2) I don't think the 3DMark03 software reviewers use reminds them to check the site

The best way to handle this would be to inform all reviewers (like with some press release) like:

Futuremark is not validating vender X's driver y results.

Of course, that's probably not a good thing for legal reasons.

I understand that Futuremark has no actual need to update their benchmark software that much more frequently. It doesn't make sense (resources, time, cost). However, something like an autoupdate system (for the software itself), but more specifically, an APPROVED DRIVERS LISTING (and possibly drivers that have failed the guidlines listing) BUILT INTO THE SOFTWARE (with an autoupdate feature) would be MORE useful than an obscure "approved driver listing on the Futuremark website".

Consider this:

If the software were to say:

Vendor X's version Y drivers failed the Futuremark guidelines (it helps to provide a reason).

The reviewers shouldn't have a problem with that. You would have to hold them to explain that to the masses (people would like to know why anyways).

Some people update their drivers frequently (also reviewers prefer to use the latest drivers in their reviews as well). It sounds "backwards" to be using drivers "one benchmark company" does not approve of to benchmark with that company's software. If the benchmark software had an autoupdatable list of approved/non-approved drivers, you may be able to prevent people from publish results that were invalid to begin with.

It does help to also give some sort of warning of a benchmark run with non-WHQL drivers. New hardware can come with non-WHQL drivers for a bit (and other reasons to have non-WHQL drivers exist) and that also needs to be pointed out to those to actually doit.

Currently the whole Futuremark guideline system is rather passive... it needs to be slightly better than it is now.

Just a thought, some sort of auto-updater for just Futuremark approved/unapproved driver info is not a lot of data. Quite a bit of people auto update a lot. For a reviewer, this would just be a small chore (smaller than visiting the Futuremark website).

 

[AzBat]

First of all, Andy great post! I agree totally. Thanks for taking the time to post your thoughts. It's nice to hear thoughts from somebody with the IHVs.

If we as an industry want to prevent the manipulation of benchmark results then it has to be done by collaboration between reviewers and the people who create the benchmarks.

I agree 100%.

My specific issue with Futuremark and how they're handling this, is that it's NOT a collaboration the way I look at it.

Though I do agree that Futuremark could be handling it better, I disagree that they're not collaborating with reviewers. How many PDFs, press releases, etc have Futuremark released in the past year? They have even made sure there has been prominent placement of this news on their home page. If reviewers are clueless that they shouldn't be using un-approved drivers when using 3DMark03, then that's their own damn fault for not keeping informed. I hold reviewers in high regard. If they're not trying to keep up with latest news regarding Futuremark, 3DMark03, etc, then they have no business reviewing products and they don't deserve me as a reader of their site.

I also disagree with your "I told you so". We've had how many sites say they would stop using 3DMark03? 2 maybe? And what sites are they? None that I had ever heard of or care about. Good riddance if you ask me. So 2 sites are not proof of your "I told you so".

They are forcing reviewers to install older driver versions in order to get "valid" scores. As I predicted earlier, this seems to be causing a few things to happen:

1) Some web sites just ignore it, and run and publish with unapproved drivers anyway
2) Some web sites are flat out refusing to use 3DMark, because of the "hassle."

Like I said earlier, only a couple of sites are doing either of those options. Even though they are, it's no big loss to Futuremark if you ask me. Any site that does one of these options IMHO is not a site I deem worthy of testing 3DMark03 or me as a visitor to their site. If and when more worthy sites start taking this action, then we and Futuremark should worry.

I want to be perfectly clear about my position on this. I want FutureMarl / 3DMark to succeed. I do agree that they are doing at least SOMETHING to combat the rampant cheating by IHVs in benchmarks, and if the Futuremark guidelines are in fact adhered to, then that would be great. And ultimately, I place the blame for this whole mess on nvidia. But Nvidia, no matter how much we (end users) bitch and moan at them, isn't going to change.

So, FutureMark should properly collaborate with reviewers, such that a solution is reached that reviewers can live with. Again, IMO, this means that FutureMark should release patches for 3DMark (when needed) after each driver release, to defeat detection / cheating.

I still believe that releasing patches for 3DMark03 is not the answer. You're already going to require reviewers to keep updated about when the new patches are released. If they can do that, then why can they not keep updated about the approved drivers? Plus, it's less for them to have to download. And what happens when you're doing comparisons before the patch is released for the new drivers? You have one set of drivers that has been adhering to the guidelines, but you have a different card that uses drivers that don't? Do you wait a couple of days for the patch to use the new drivers, thus punishing the card that already had approved drivers? Or do you go ahead and use older drivers that are approved that way both cards results will be available at the time of your deadline? Then what do you do later when they do finally release a patch? Most of the time you have to send the card back. So you may not have one or both used in the comparison. It makes more sense to keep the number of patches to a minimum. There will be even more headaches if you require reviewers to have to wait for new patches before you can test with the latest drivers.

I do however disagree with Worm that changing drivers doesn't take very much time. Whenever I tested hardware I would start from a clean install of Windows when changing drivers. I formatted the drive, installed Windows and all patches and service packs, then installed the testing apps(and their patches), then created an image using DriveImage. Next I would install the drivers I wanted to test with, then run the apps. If I wanted to install different drivers or use a different card, then I would restore the image, reboot and install the new drivers. I sure hope that reviewers today are not testing the card with the latest drivers installed, then just remove them and install the older ones without doing a process similar to mine. There is so much stuff that could be still left in the directories or registry that could cause conflicts or taint any results gathered.

It's the obligation of the reviewers to download the latest 3DMark patch, just as they are "obligated" to download the latest drivers from the IHVs.

Of course, this is more work for FutureMark. But I feel it is the ONLY REASONABLE WAY that they can properly enforce their guidelines.

I 100% TOTALLY DISAGREE! It's not reasonable WHATSOEVER for Futuremark to be required to have to patch their software every time a IHV decides to release new cheating drivers.

Without doing this, I just see more and more review sites not bothering with it...which again is a shame because it is a great tool.

I'd rather that sites just do their goddamn job and quit they're damn belly-aching. Bunch of pansies and all they're whining that it's "too hard" to keep up with the approved driver list. Boo hoo. If they give any of your reasons for not using 3DMark03, then they're not a site I trust doing quality reviews and I won't visit them, period!

Tommy McClain

 

[AzBat]

]

It's the obligation of the reviewers to download the latest 3DMark patch, just as they are "obligated" to download the latest drivers from the IHVs.

So what you are saying is that downloading a new patch for 3DMark (& installing it) everytime a new driver (from any IHV) is released would be less hassle than simply installing the drivers we have approved? I am not sure if I see the logic in that. :? Or then my head simply isn't working well (terrible flu.. ).

I completely understand his logic. He's looking at it from the reviewers point of you. The lazy reviewer that is. He doesn't want them to have to think or do their job correctly. He just wants them to be able to run the latest driver without having to think if the driver is approved or not. Sounds great doesn't it? That's definitely the kind of reviewer I trust. NOT!

Sorry Joe for picking on you, but you're just insane!!

Tommy McClain

 

[Mariner]

What is needed is a pop up window appearing on screen which appears every time you run 3DMark. This should state something like:

"Please ensure that you have installed the latest version of 3DMark available from (URL) and check that you are using the approved drivers for your card available from (another URL). Please note that scores produced by unapproved drivers are not valid blah blah etc etc..."

If you have to read something like this and then a click on a button is required to remove this from the screen and start the benchmark, there are no excuses for not knowing which drivers are valid or not. Have the message in the pop-up window flashing in BIG RED LETTERS AS WELL to make sure users get the point!

 

[AzBat]

Yes. A patch doesn't require a system rebuild (as changing drivers would to maintain fair results). A patch number makes for easy user comparison, if they see an old version number then they will complain.

For me a new patch does mean updating my system image to include the new patch. I would restore the image, then install the patch and then make a new image of the drive before I ever used the software to test. Call it more work, but you can't say it doesn't produce valid results. I think today's reviewers are becoming a bunch of lazy bastards.

Tommy McClain

 

[Mariner]

What is needed is a pop up window appearing on screen which appears every time you run 3DMark. This should state something like:

"Please ensure that you have installed the latest version of 3DMark available from (URL) and check that you are using the approved drivers for your card available from (another URL). Please note that scores produced by unapproved drivers are not valid blah blah etc etc..."

If you have to read something like this and then a click on a button is required to remove this from the screen and start the benchmark, there are no excuses for not knowing which drivers are valid or not. Have the message in the pop-up window flashing in BIG RED LETTERS AS WELL to make sure users get the point!

 

[AzBat]

Wow, looks like I'm going to be agreeing with demalion. Probably has a lot to do with actually understanding his message and/or not giving up reading it half way through. Hehe Sorry demalion, just being funny(or at least trying).

Where I depart from Joe most distinctly is the idea that producing continual patches is feasible for Futuremark.

The process, AFAICS:

• costs time and money (as it does nVidia to continue to cheat, but nVidia has more resources)
• is of limited benefit (due to nVidia persisting in defeating it again in the next driver version that they will release anyways)
• provides a perpetual strawman for nVidia to attack in terms of blame (not a logical one, but an effective one to those uninformed about the technical issues)

This is a losing proposition in practical terms, to the best of my knowledge.

I agree on all points. Couldn't say it any better.

Where I agree is in the idea that Futuremark is failing to take certain steps that do seem to be within their means and are not losing propositions:

• Failing to deal with the PS 2.0 test was a significant mistake, because it throws away a test (this doesn't mean it wasn't an oversight, or even understandable, it just means it is a significant point of failure that weakens credibility).
• There are steps Futuremark could indeed take and information they could make more accessible that would markedly improve the success of what they did do...not because it is their fault that these steps are necessary, or their fault that some reviewers are irresponsible, but because their goal should be to succeed without concern of blame. To clarify, it is akin to advertising...for a good company trying to make a good product, it isn't just a matter of making the good product and suffering when inferior products sell better because of more advertising, it is a matter of taking all the available opportunities that are within your means to illustrate how your product is a good product. The ORB information ideas that have persistently been presented, and even the "watermark" idea as recently repeatedly proposed by dw, are indeed silly when evaluated by the criteria of the information already being provided, but not at all silly, and even essential, from the perspective of opportunities Futuermark has available to them to get the message out, and are passing up.

Wow, great insight! I agree. I've always stated that Futuremark could possibly do more to make the process work better. And it is un-nerving to hear that they are reluctant to consider some of these ideas or feel that some of them are not necessary. It's like they're denying that they have a problem. Maybe they need to join a 12-step program.

The continued problem, AFAICS, is that Futuremark is caught up in the blame game nVidia has tried to define this situation as, and have lost sight of the larger perspective of their relationship to their users outside of the situation nVidia is working to define. Their responses seem to have been primarily geared at reactively dealing with things within the framework of blame in which they were attacked, and not taking the best advantage of the other avenues available to them outside of that framework.

OK, I did get through you whole message and if I understand you correctly, then I agree as well. They definitely seem to be hung up something that they shouldn't be.

Tommy McClain

 

[Nick[FM]]

I do however disagree with Worm that changing drivers doesn't take very much time. Whenever I tested hardware I would start from a clean install of Windows when changing drivers. I formatted the drive, installed Windows and all patches and service packs, then installed the testing apps(and their patches), then created an image using DriveImage. Next I would install the drivers I wanted to test with, then run the apps. If I wanted to install different drivers or use a different card, then I would restore the image, reboot and install the new drivers. I sure hope that reviewers today are not testing the card with the latest drivers installed, then just remove them and install the older ones without doing a process similar to mine. There is so much stuff that could be still left in the directories or registry that could cause conflicts or taint any results gathered.

Isn't it easier to install Windows, install all updates, DX etc. and then make a ghost image of it for future use? Then when you need to benchmark with a clean system, simply load the image (takes what, a couple of minutes? Maybe 5), install the disp drivers (takes less than a couple of minutes), install the benchmark (possibly update it if you haven't got the latest full install) and run it. When you want to change drivers or whatever and don't trust the driver uninstallers, simply load the image again and install the drivers & benchmark util. This operation doesn't take that long. Trust me, I do it very often! The clever ones load the clean image once in a while and update the Windows updates & mobo/sound/whatever (except disp drv) drivers as and makes a new image of it to keep it up-to-date. But then again, someone might find that very annoying and time consuming. I am used to it, so I don't have any problems with it.

 

[AzBat]

What is needed is a pop up window appearing on screen which appears every time you run 3DMark. This should state something like:

"Please ensure that you have installed the latest version of 3DMark available from (URL) and check that you are using the approved drivers for your card available from (another URL). Please note that scores produced by unapproved drivers are not valid blah blah etc etc..."

If you have to read something like this and then a click on a button is required to remove this from the screen and start the benchmark, there are no excuses for not knowing which drivers are valid or not. Have the message in the pop-up window flashing in BIG RED LETTERS AS WELL to make sure users get the point!

Excellent suggestion. Unfortunately any kind of update like this is going to require Futuremark to release a new patch for 3DMark03. I'm of the opinion, the current patch should be the last Futuremark should ever release for 3DMark03. They need to take these ideas(pop-up windows, auto-updates, etc) and put them to use in the next version of 3DMark ASAP. The quicker they release the next version, the better for everybody involved.

Tommy McClain

 

[AzBat]

]

I do however disagree with Worm that changing drivers doesn't take very much time. Whenever I tested hardware I would start from a clean install of Windows when changing drivers. I formatted the drive, installed Windows and all patches and service packs, then installed the testing apps(and their patches), then created an image using DriveImage. Next I would install the drivers I wanted to test with, then run the apps. If I wanted to install different drivers or use a different card, then I would restore the image, reboot and install the new drivers. I sure hope that reviewers today are not testing the card with the latest drivers installed, then just remove them and install the older ones without doing a process similar to mine. There is so much stuff that could be still left in the directories or registry that could cause conflicts or taint any results gathered.

Isn't it easier to install Windows, install all updates, DX etc. and then make a ghost image of it for future use? Then when you need to benchmark with a clean system, simply load the image (takes what, a couple of minutes? Maybe 5), install the disp drivers (takes less than a couple of minutes), install the benchmark (possibly update it if you haven't got the latest full install) and run it. When you want to change drivers or whatever and don't trust the driver uninstallers, simply load the image again and install the drivers & benchmark util. This operation doesn't take that long. Trust me, I do it very often! The clever ones load the clean image once in a while and update the Windows updates & mobo/sound/whatever (except disp drv) drivers as and makes a new image of it to keep it up-to-date. But then again, someone might find that very annoying and time consuming. I am used to it, so I don't have any problems with it.

Worm,

I think you're too sick to be chatting. I explained I did the exact same thing. I just used PowerQuest's DriveImage software at the time. Never was a fan of Norton. Go back and read my message and subsequent ones again.

Tommy McClain

 

[Nick[FM]]

Worm,

I think you're too sick to be chatting. I explained I did the exact same thing. I just used PowerQuest's DriveImage software at the time. Never was a fan of Norton. Go back and read my message and subsequent ones again.

Tommy McClain

I somehow got the impression that you would format & install Windows everytime you want to install new drivers. Heh.. But anyway, restoring an image, installing a set of drivers and a benchmark doesn't take that long. As I said, I do it almost every day and if you know what you are doing, it goes pretty quickly and without any hassle. At least for me.