Optimization Guidelines QA

[nelg]

To clarify Worm, will Futuremark be using the expertise of other members to help determine if certain driver behaviors are to considered valid optimizations or cheats ? Would allowing other members to help investigate violate any NDA’s ?

 

[digitalwanderer]

Would allowing other members to help investigate violate any NDA’s ?

Do you think an NDA would really even slow them down for the investigating bit of it?

 

[Nick[FM]]

To clarify Worm, will Futuremark be using the expertise of other members to help determine if certain driver behaviors are to considered valid optimizations or cheats ? Would allowing other members to help investigate violate any NDA’s ?

We will of course allow anyone (individuals and companies) to submit reports or findings to us. We encourage everyone to do that. We are also looking for new methods of making our software more proof against any kind of detection etc. but hopefully we don't have to do that anymore. You will understand the whole process as soon as we publish it. ATM it looks like it is a very good and solid solution, but we want that all our members are with us on that.

 

[just me]

"In general, we can only approve WHQL'ed drivers that we have verified in-house, but we reserve the right to approve non-WHQL drivers in special case-by-case scenarios. ..."

Thank you! Since there aren't going to be any more 98/98SE/Me drivers WHQL'd: I hope you consider testing Omega's offerings for 98-Me drivers as there is still a significant amount of 98-Me users in the world.

Question: Is FM now going to detect 3DM2K1 & PCMark drivers to comply with the 'All 3DM products' statement?

Enjoy your Friday night. 8)

 

[Nick[FM]]

Ok one more, then I'm off..

Thank you! Since there aren't going to be any more 98/98SE/Me drivers WHQL'd: I hope you consider testing Omega's offerings for 98-Me drivers as there is still a significant amount of 98-Me users in the world.

The Omega drivers are (AFAIK) not WHQL'd. That means that we won't test them, unless a special case scenario. :?

Question: Is FM now going to detect 3DM2K1 & PCMark drivers to comply with the 'All 3DM products' statement?

Enjoy your Friday night. 8)

The guidelines are for our supported 3DMark series only. The versions we currently support are 3DMark2001 SE and 3DMark03.

I sure will enjoy my Friday night. You can bet on that!

 

[Joe DeFuria]

]The Omega drivers are (AFAIK) not WHQL'd. That means that we won't test them, unless a special case scenario. :?

Just like to say that's completely reasonable and understandable. You can't be expected to test every 3rd party "hacked" drivers or beta / leaked driver out there.

Futuremark should limit testing (and therefore published ORB results) to WHQL drivers only, and furthermore only those WHQL drivers that also pass FMQL. (Which is exactly what they're doing.)

 

[dan2097]

Great Q&A Worm

Thanks alot for answering my question about when these new guidelines will be introduced

If you stick to those guidelines + the answers in the Q&A then there doesn't seem to be any loopholes for perverting the validity of the benchmark

 

[dan2097]

Futuremark should limit testing (and therefore published ORB results) to WHQL drivers only, and furthermore only those WHQL drivers that also pass FMQL. (Which is exactly what they're doing.)

Or if they dont want to effectively wipe the browsable datebase they could have FMQL drivers /all drivers just as they have with WHQL at the moment i.e. FMQL drivers defaultly used for comparison but an option to search through all scores. Of course only FMQL scores would be used in any official comparisons/the hall of fame

 

[Mark]

It's a great step, and I'm very pleased with the guidelines and Q&A document in general. I just hope these rules and guidelines don't turn out to be kin to a pro-wrestling referee; good for putting on a show of authority, but easily tossed aside when they get in the way.

 

[cthellis42]

]

The Det50 drivers are currently using "quasi trilinear" filtering in all Direct3D programs.

We will only test and approve WHQL drivers that are going public. The Det50 series aren't officilly released yet, so let's see about them when the time comes.

Would this be one of those situations where you would allow them to, say, detect 3DMark to remove said broadscale "quasi trilinear" filtering methods and utilize one consistent to FutureMark's guidelines?

 

[AzBat]

I'm going to make my reply to the QA that Futuremark just released. In order to make sure I'm consistent I'm including and replying to my previous comments before I get to comments from others. This will probably end up being rather long. I'll try to keep it readable.

AzBat[/url]]

AzBat[/url]]All I've got to say is that come September 19, Futuremark had better be acknowledging that NVIDIA and its drivers not living up to your rules and guidelines and that their scores will not be supported. If you end-up releasing a crap set of rules and then don't have the back-bone to enforce them, then be ready to pack-up your office belongings and find another job because I don't think your company will be around much longer.

I first would like to say that I believe these rules are a step in the right direction. They have the potential to give back Futuremark their integrity and relevance to their benchmark. So in that regard I applaud them for making these rules. Are they the best? No. Are they the worst? No, not by a long shot. But like I said earlier, rules by themselves are not going to save them. Enforcement and clarification is needed.

I was glad to see that they understood that enforcement and clarfication was greatly needed. Bravo. Since enforcement was the first questions they answered I'm sure they received quite a few questions about it. From the questions they answered with regards to enforcement, I'm satisfied and agree with their answers. I'm sure they did their homework on this and have a plan that should bring integrity and validity back to their benchmark.

AzBat[/url]]Rule 2: I like the first part of the rule since it states "indirectly". However, I don't like the idea of the "but...". Will the approval process and what is approved(IHV and the detection needed) be made public? What's considered a hardware error?

Looks like they answered these questions to my satisfaction.

AzBat[/url]]Rule 3: If I understand this right, this means you can't look at the benchmark and then create clipping planes based on what was rendered on screen. Same goes for the PrtScrn issue. Bravo.

Looks like I understood the rule correctly. Good deal.

AzBat[/url]]Rule 4: I like this one as well. Basically if you make an optimization in your driver, you better make sure as hell it works on other applications that use the same thing and that there is no difference in the output. Unfortunately they left a loop hole by saying "mathematically consistent". I'm sure they got some grief for the wording here. They probably used the word "correct" instead, but then had to change it because there are no correct definitions. The reference rasterizer should be correct, but then again isn't it based on some IHV's hardware? If your hardware wasn't used as the reference rasterizer, then I could see how other IHVs might have a problem with it.

Looks like they chose a better word this time, "equivalent". Their answer seems to close the loop hole a little better, but I still think there is some wiggle room there. Guess we will just have to wait and see.

AzBat[/url]]All in all, I think these rules explain in enough detail of how IHVs are suppose to act. Unfortunately, I initially see some of them trying to test to see if or how Futuremark enforces the rules. That brings up my biggest question, how are Futuremark going to enforce them? Do they now have some kind of process that beta members are now required to follow? What about companies that are not members? Does Futuremark now have a process similar to WHQL? Meaning, are they are now required to send Futuremark drivers for testing before they can be approved? Is Futuremark going to make public which public driver sets are approved and which ones are not? Will they state why they weren't approved? Or are they going to do something totally different and use help from the press or beta members to help them do the policing?

Looks like they answered all of these questions. Impressive.

AzBat[/url]]There are lot of questions still left that I believe need answering. I also would like to see official responses from the beta members. Does NVIDIA plan on playing by the rules?

Still haven't seen any responses from any of the members. I sure would love to at least see some kind of official article from B3D on this matter. Would be interesting to see what(if any) suggestions they made made it into the rules and QA.

AzBat[/url]]Now, my last concern is what happens if and when drivers are found to violate these rules? Will the results just not be included in the ORB results or will there be other repercussions? Will the violations be publicly made? Also, are these rules legally binding? Will they be included in the license agreement? I don't want a IHV to violate the rules and then state they're not legally required to do so because of some kind of loop hole.

Looks like their answers to these questions have put my mind a little at ease. However, they didn't touch on my legal concerns. Worm might have in some of his responses and if so I'll follow up on it.

AzBat[/url]]

3) How will 3DMark "police" their guidelines? Rely on 3rd party investigations...do their own set of tests....combination of both?

Agreed. This is one of my biggest questions. I'd rather Futuremark have an official approval process that's made public and not rely on 3rd parties for the policing.

Looks like they agreed. However, looks like they will have some kind of "cheater" hotline to tell them of questionable optimizations. Cool. Hopefully they won't rely on this totally.

AzBat[/url]]I would love for Futuremark to make public the approved detections(if any), but I'm not sure if IHVs would allow Futuremark to announce any of them. Announcing the "hardware errors" could potentially scare away customers that see the hardware as being defective. I could see the potential for some hardware having glitches that requires some special casing in the drivers, but not being so bad that the hardware is useless for the tasks it was purchased for. But yes, it would be nice to keep the customer informed so they know what they're getting into.

I didn' see mention in the QA if they would announce the approved detections, but I think I recall Worm following up on this. If so, I'll post comments on it later.

AzBat[/url]]In closing, I would like to say that these new rules show me that Futuremark is finally seeing that they need to protect 3DMark from cheating. Unfortunately they don't go into any details on how they expect to enforce these new rules. Nor do they say if any IHVs/drivers violate currently violate those rules. I believe once we hear more details on those issues we will have a better idea on whether 3DMark and Futuremark have regained the trust and integrity that is needed to be the industry leading benchmark. You can be sure I'll voice my opinion once they do.

Futuremark definitely made some progress with their QA. I'm very pleased with them. I wasn't too surprised to hear that they haven't approved of any current drivers. So in effect, all IHVs are all on the same page until Oct. 31st. The 35 day wait is understandable and agreeable. Like others have said provided Futuremark doesn't push it back farther. Using their previous track record I can see it happening though. If they do, then I'm not going to be very pleased at all.

Using Joe's sig as template this is how I see Futuremark...

Before rules and QA
Integrity/Validity[-------|----X--]No Integrity/Validity

After rules and QA
Integrity/Validity[------X|-------]No Integrity/Validity

Once Oct. 31 gets here Futuremark has the ability to raise the Integrity/Validity meter much higher and approach the levels before the cheating. Considering their strong rules and clarifications I would be stupid not to give them the benefit of the doubt. So with that said they've pretty much dug themselves out the hole in my eyes. I just hope they don't jump back in it. There are still a couple of loose ends, but this is not the time to get into them. Futuremark deserve the accolades they're getting today and I'm not going to ruin it for them.

Tommy McClain

 

[AzBat]

After my first pass reading through it:

Nice smiley.

Just a few questions:

Q: Have you been discussing the new ground rules with all members in the benchmark development program, and have they all approved them?
A: We have worked with all members in drafting these guidelines and we have solicited feedback from them throughout the process. We have gotten very strong agreement to these rules from an overwhelming majority of the members.

So, um, care to mention names of those in the "minority?"

Hehehe. I was thinking the same thing.

We presume that manufacturers will want to make sure the validity of their drivers with us before they ship those to reviewers.

I think that's an optimisitc presumption. You may want to include in your terms of use, that those publishing scores (in PR releases, reviews, etc.) on non FutureMark approved drivers, must indicate such.

I totally agree. They need to make sure that these new rules are legally binding and are a part of the Terms of Use and License Agreement.

Tommy McClain

 

[K.I.L.E.R]

Funny how there are no guidelines on users.

I could cheat my ass off running 3dmark03 rendering in wireframe mode. I could get away with it too assuming the score isn't too far fetched. What's to stop me from doing that?

 

[cthellis42]

I could cheat my ass off running 3dmark03 rendering in wireframe mode. I could get away with it too assuming the score isn't too far fetched. What's to stop me from doing that?

Um, nothing I suppose. What's to stop anyone from just outright LYING about the numbers they benchmark? Nothing either. But what does one user's voice matter? Not much.

Hardware reveiwers could be a problem, sure, but reputable ones got their reputations by existing within the bounds of the industry as it effects their readership (we average users up to enthusiasts), so I kind of expect anyone with a reputation built one way to continue the same way. 'sides, if results are too far off from the general concensus, they'll be looked at skeptically unless backed up by lots of explanation and documentation, and if they get caught at blatant cheating...? Well that's one credability hit I don't think any of them are willing to take the chance on.

 

[just me]

Funny how there are no guidelines on users.

I could cheat my ass off running 3dmark03 rendering in wireframe mode. I could get away with it too assuming the score isn't too far fetched. What's to stop me from doing that?

Try it & see what happens. 3DAnalyze & a few other proggies have been prohibited from publishing for some time & when 'suspect' scores are found by users > they are investigated .... have been for quite awhile too. 8)

 

[K.I.L.E.R]

Funny how there are no guidelines on users.

I could cheat my ass off running 3dmark03 rendering in wireframe mode. I could get away with it too assuming the score isn't too far fetched. What's to stop me from doing that?

Try it & see what happens. 3DAnalyze & a few other proggies have been prohibited from publishing for some time & when 'suspect' scores are found by users > they are investigated .... have been for quite awhile too. 8)

When I get better at my programming I'm going to make it a hobby to dismantle 3DAnalyze along with other benchmark progs to see how far I can get cheated scores through to publishing.
Those cheated scores will be believeable as well.

 

[WaltC]

Worm, I'm glad to see you guys addressing the enforcement issue. I still have a couple of questions, though...

You guys mentioned awhile back that you would be considering implementing vendor-specific code paths in the benchmark itself going forward. Have you reached a decision on that yet? It strikes me that if you were to begin allowing such code paths, then there'd be scant reason for attempting to cheat the benchmark in the drivers. I'm assuming your answer is that "We have decided not to allow any specific IHV code paths in the software." Hope I'm correct...

Regarding enforcement of your rules, you demonstrated admirably what you were capable of doing a few months back when you issued your audit report and issued a recompilation patch of the software, which made driver detection much more difficult. As well as publishing the audit report you disallowed certain drivers. It seems to me going forward that you might consider quarterly recompilation patches of the software on a regular basis designed to defeat driver detection, as this might actually be an easier way for you to block it without any direct confrontation with your IHV partners, which I know you'd like to avoid. Do you plan to issue such patches on a regular basis going forward? Just wondering...

Thanks for the info...

 

[Myrmecophagavir]

]

The Det50 drivers are currently using "quasi trilinear" filtering in all Direct3D programs.

We will only test and approve WHQL drivers that are going public. The Det50 series aren't officilly released yet, so let's see about them when the time comes.

Would this be one of those situations where you would allow them to, say, detect 3DMark to remove said broadscale "quasi trilinear" filtering methods and utilize one consistent to FutureMark's guidelines?

I don't think that would be in the spirit of the guidelines, since then 3DMark would not be representative of game performance which is its goal.

 

[Nick[FM]]

You guys mentioned awhile back that you would be considering implementing vendor-specific code paths in the benchmark itself going forward. Have you reached a decision on that yet? It strikes me that if you were to begin allowing such code paths, then there'd be scant reason for attempting to cheat the benchmark in the drivers. I'm assuming your answer is that "We have decided not to allow any specific IHV code paths in the software." Hope I'm correct...

As of yet we haven't made any decicions concerning this. We are still working out all possible (and impossible) details we need to know about vendor-specific code paths. I don't know when we have the plans done, but I personally hope that we can let you guys know about them as soon as we have something solid and concrete to report.

Regarding enforcement of your rules, you demonstrated admirably what you were capable of doing a few months back when you issued your audit report and issued a recompilation patch of the software, which made driver detection much more difficult. As well as publishing the audit report you disallowed certain drivers. It seems to me going forward that you might consider quarterly recompilation patches of the software on a regular basis designed to defeat driver detection, as this might actually be an easier way for you to block it without any direct confrontation with your IHV partners, which I know you'd like to avoid. Do you plan to issue such patches on a regular basis going forward? Just wondering...

Good question. Currently we have a very good and possible plan on paper, but we are always looking into new and improved ways to be "on top of things". What we will do concerning the patches you mentioned is still not set. I think it applies here too, that when the process will be posted, many of your questions will be answered. At least I hope so!

 

[Nick[FM]]

Funny how there are no guidelines on users.

I could cheat my ass off running 3dmark03 rendering in wireframe mode. I could get away with it too assuming the score isn't too far fetched. What's to stop me from doing that?

We have some "recommended testing procedure" guidelines for users too. Have been there for a long time. Here are a couple of examples:

Set all your display properties settings to "Default" under Direct3D
Close any open applications and background programs

More can be found here.