Optimization Guidelines QA

To clarify Worm, will Futuremark be using the expertise of other members to help determine if certain driver behaviors are to considered valid optimizations or cheats ? Would allowing other members to help investigate violate any NDA’s ?

 

Do you think an NDA would really even slow them down for the investigating bit of it?

 

"In general, we can only approve WHQL'ed drivers that we have verified in-house, but we reserve the right to approve non-WHQL drivers in special case-by-case scenarios. ..."

Thank you! Since there aren't going to be any more 98/98SE/Me drivers WHQL'd: I hope you consider testing Omega's offerings for 98-Me drivers as there is still a significant amount of 98-Me users in the world.

Question: Is FM now going to detect 3DM2K1 & PCMark drivers to comply with the 'All 3DM products' statement?

Enjoy your Friday night. 8)

 

Just like to say that's completely reasonable and understandable. You can't be expected to test every 3rd party "hacked" drivers or beta / leaked driver out there.

Futuremark should limit testing (and therefore published ORB results) to WHQL drivers only, and furthermore only those WHQL drivers that also pass FMQL. (Which is exactly what they're doing.)

 

Great Q&A Worm

Thanks alot for answering my question about when these new guidelines will be introduced

If you stick to those guidelines + the answers in the Q&A then there doesn't seem to be any loopholes for perverting the validity of the benchmark

 

Or if they dont want to effectively wipe the browsable datebase they could have FMQL drivers /all drivers just as they have with WHQL at the moment i.e. FMQL drivers defaultly used for comparison but an option to search through all scores. Of course only FMQL scores would be used in any official comparisons/the hall of fame

 

It's a great step, and I'm very pleased with the guidelines and Q&A document in general. I just hope these rules and guidelines don't turn out to be kin to a pro-wrestling referee; good for putting on a show of authority, but easily tossed aside when they get in the way.

 

Would this be one of those situations where you would allow them to, say, detect 3DMark to remove said broadscale "quasi trilinear" filtering methods and utilize one consistent to FutureMark's guidelines?

 

I'm going to make my reply to the QA that Futuremark just released. In order to make sure I'm consistent I'm including and replying to my previous comments before I get to comments from others. This will probably end up being rather long. I'll try to keep it readable.

I was glad to see that they understood that enforcement and clarfication was greatly needed. Bravo. Since enforcement was the first questions they answered I'm sure they received quite a few questions about it. From the questions they answered with regards to enforcement, I'm satisfied and agree with their answers. I'm sure they did their homework on this and have a plan that should bring integrity and validity back to their benchmark.

Looks like they answered these questions to my satisfaction.

Looks like I understood the rule correctly. Good deal.

Looks like they chose a better word this time, "equivalent". Their answer seems to close the loop hole a little better, but I still think there is some wiggle room there. Guess we will just have to wait and see.

Looks like they answered all of these questions. Impressive.

Still haven't seen any responses from any of the members. I sure would love to at least see some kind of official article from B3D on this matter. Would be interesting to see what(if any) suggestions they made made it into the rules and QA.

Looks like their answers to these questions have put my mind a little at ease. However, they didn't touch on my legal concerns. Worm might have in some of his responses and if so I'll follow up on it.

Looks like they agreed. However, looks like they will have some kind of "cheater" hotline to tell them of questionable optimizations. Cool. Hopefully they won't rely on this totally.

I didn' see mention in the QA if they would announce the approved detections, but I think I recall Worm following up on this. If so, I'll post comments on it later.

Futuremark definitely made some progress with their QA. I'm very pleased with them. I wasn't too surprised to hear that they haven't approved of any current drivers. So in effect, all IHVs are all on the same page until Oct. 31st. The 35 day wait is understandable and agreeable. Like others have said provided Futuremark doesn't push it back farther. Using their previous track record I can see it happening though. If they do, then I'm not going to be very pleased at all.

Using Joe's sig as template this is how I see Futuremark...

Before rules and QA
Integrity/Validity[-------|----X--]No Integrity/Validity

After rules and QA
Integrity/Validity[------X|-------]No Integrity/Validity

Once Oct. 31 gets here Futuremark has the ability to raise the Integrity/Validity meter much higher and approach the levels before the cheating. Considering their strong rules and clarifications I would be stupid not to give them the benefit of the doubt. So with that said they've pretty much dug themselves out the hole in my eyes. I just hope they don't jump back in it. There are still a couple of loose ends, but this is not the time to get into them. Futuremark deserve the accolades they're getting today and I'm not going to ruin it for them.

Tommy McClain

 

Nice smiley.

Hehehe. I was thinking the same thing.

I totally agree. They need to make sure that these new rules are legally binding and are a part of the Terms of Use and License Agreement.

Tommy McClain

 

Funny how there are no guidelines on users.

I could cheat my ass off running 3dmark03 rendering in wireframe mode. I could get away with it too assuming the score isn't too far fetched. What's to stop me from doing that?

 

Um, nothing I suppose. What's to stop anyone from just outright LYING about the numbers they benchmark? Nothing either. But what does one user's voice matter? Not much.

Hardware reveiwers could be a problem, sure, but reputable ones got their reputations by existing within the bounds of the industry as it effects their readership (we average users up to enthusiasts), so I kind of expect anyone with a reputation built one way to continue the same way. 'sides, if results are too far off from the general concensus, they'll be looked at skeptically unless backed up by lots of explanation and documentation, and if they get caught at blatant cheating...? Well that's one credability hit I don't think any of them are willing to take the chance on.

 

Try it & see what happens. 3DAnalyze & a few other proggies have been prohibited from publishing for some time & when 'suspect' scores are found by users > they are investigated .... have been for quite awhile too. 8)

 

When I get better at my programming I'm going to make it a hobby to dismantle 3DAnalyze along with other benchmark progs to see how far I can get cheated scores through to publishing.
Those cheated scores will be believeable as well.

 

Worm, I'm glad to see you guys addressing the enforcement issue. I still have a couple of questions, though...

You guys mentioned awhile back that you would be considering implementing vendor-specific code paths in the benchmark itself going forward. Have you reached a decision on that yet? It strikes me that if you were to begin allowing such code paths, then there'd be scant reason for attempting to cheat the benchmark in the drivers. I'm assuming your answer is that "We have decided not to allow any specific IHV code paths in the software." Hope I'm correct...

Regarding enforcement of your rules, you demonstrated admirably what you were capable of doing a few months back when you issued your audit report and issued a recompilation patch of the software, which made driver detection much more difficult. As well as publishing the audit report you disallowed certain drivers. It seems to me going forward that you might consider quarterly recompilation patches of the software on a regular basis designed to defeat driver detection, as this might actually be an easier way for you to block it without any direct confrontation with your IHV partners, which I know you'd like to avoid. Do you plan to issue such patches on a regular basis going forward? Just wondering...

Thanks for the info...

 

I don't think that would be in the spirit of the guidelines, since then 3DMark would not be representative of game performance which is its goal.