We just posted a new pdf about the enforcement of the optimization guidelines:
http://www.futuremark.com/companyinfo/Enforcement_Process.pdf
http://www.futuremark.com/companyinfo/Enforcement_Process.pdf
Process to Enforce 3DMark® Optimization Guidelines
- Thread starter Nick[FM]
- Start date
.pdf said:
If I understand this right, Futuremark informs BDP members of mentioned issues before contacting the manufacturer about it?
Doesn't that raise potential problems?
What if a "legitimate" bug finds its way into the drivers that reveals a confidential aspect of HW/driver technology to competing companies?
OK, I'm down to calling bugs legitimate. what next?
Joe DeFuria
Legend
vb said:
??
It doesn't seem like it's BEFORE contacting the manufacturer at all:
Futuremark said:
The manufacturer is contacted immediately.
Though it's not clear to me when exactly the BDP members are contacted in the process. At the same time that the manufacturer is? Or after the 1 week deadline for the manufacturer's response?
Based on the block diagram, it looks like the BDP members and the manufacturer are informed about potential issues at the same time. This is reasonable, as long as the BDP members are also kept in the immediate loop on progress made to the resolution on the issue. (Which appears to be the case. That is, when the manufacturer gives the response to FM, FM passes that information on to the BDP.
Informing BDP and the manufacturer at the same time puts some pressure on the manufacturer to respond more quickly with some response, which is a good thing.
The obvious question that I was hoping would be answered at this time is: what's the sate of the current WHQL drivers from all vendors and their products?
Which raises a related question: drivers behanve differently depending on the product using them. It's entirely feasible, if not likely, that one piece of hardware and driver X may "pass" FutureMark tests, while a different piece of hardware with the SAME driver could fail. Is FutureMark preparing the patch / driver "review" database with this in mind?
In other words, it's not enough to label a driver version as "reviewed" or not. It has to be a combination of driver / chip-level hardware.
This is definately a step in the right direction, and if I may say- has been a long time coming.
Discounting selective discretion with what are deemed "valid drivers" and to what fuzzy degree they adhere to given benchmark driver guidelines, this whole thing now boils down to two issues-
1) ORB/FutureMark website results (which this PDF outlines, but fails to mention if such changes will be retroactive.. i.e. does it only effect newly attempted published scores or will previous scores on drivers deemed improper be purged from the database?)
2) Website/3rd party usage of the benchmark when using 3DMark scores for product comparisons in the context of website reviews.
Issue #2 is not compensated nor handled by anything I've seen so far, and is actually the primary problem or concern. I'm not sure what market research has been performed, but I don't see IHV hardware purchase decisions/comparisons having much reliance on ORB score results. ORB results are generally from people with the already-purchased hardware, so I can easily see any IHV being satisfied with a simple repercussion of "you can't publish cheating scores"... just as long as Website or Magazine A,B and C will continue to publish graphs with the overly inflated and inaccurate score comparisons to sell units.
The true solution is to advance the product to fail to deliver scores on uncertified drivers. A simple WHQL certification check would be the best first pass, and with some form of downloadable "certified" driver system (much like Antivirus software definitions) to allow the product to even produce a score. Obviously such a change would be a long term goal, but such discussion/dialogue on such a future would more clearly show the motives of FutureMark and the interest of the accuracy of their products.
Discounting selective discretion with what are deemed "valid drivers" and to what fuzzy degree they adhere to given benchmark driver guidelines, this whole thing now boils down to two issues-
1) ORB/FutureMark website results (which this PDF outlines, but fails to mention if such changes will be retroactive.. i.e. does it only effect newly attempted published scores or will previous scores on drivers deemed improper be purged from the database?)
2) Website/3rd party usage of the benchmark when using 3DMark scores for product comparisons in the context of website reviews.
Issue #2 is not compensated nor handled by anything I've seen so far, and is actually the primary problem or concern. I'm not sure what market research has been performed, but I don't see IHV hardware purchase decisions/comparisons having much reliance on ORB score results. ORB results are generally from people with the already-purchased hardware, so I can easily see any IHV being satisfied with a simple repercussion of "you can't publish cheating scores"... just as long as Website or Magazine A,B and C will continue to publish graphs with the overly inflated and inaccurate score comparisons to sell units.
The true solution is to advance the product to fail to deliver scores on uncertified drivers. A simple WHQL certification check would be the best first pass, and with some form of downloadable "certified" driver system (much like Antivirus software definitions) to allow the product to even produce a score. Obviously such a change would be a long term goal, but such discussion/dialogue on such a future would more clearly show the motives of FutureMark and the interest of the accuracy of their products.
Compliments to FutureMark for putting it out in a timely manner and my thanks to Worm for coming around and telling us 'bout it. 
I still haven't actually READ it yet though... (I hate .pdfs! Where is the .html version? )
I still haven't actually READ it yet though... (I hate .pdfs! Where is the .html version?
)Patric Ojala
Newcomer
vb said:
Hmm, I must obvioulsy read wörms .pdf again
The idea is to first contact the driver manufacturer.
What about flogging, did you include flogging in the punishments? 
worm[Futuremark said:
Worm,
Thanks for the link and the heads up! I just checked the PDF and have to say that I don't have much criticisms, if any, so far of your actual process. Good job. Unfortunately, there is one thing I have a problem with...
Joe DeFuria said:
Agreed. I'm a little disappointed that the Oct. 31st date wasn't adhere to for notifying us which drivers are certified for use with 3DMark. In my last post on this topic...
http://www.beyond3d.com/forum/viewtopic.php?t=8157&start=40
I stated...
AzBat said:
Looks like they haven't pushed the date back per se, but they did say they would release a new build of 3DMark. Unfortunately, they give no timeline for the release of the new build or the list of drivers that pass the certification. I know things take time, but the guidelines specifically said they would "begin full-fledged enforcing at latest on October 31st.". Nothing in the PDF today tells me if they're actually doing that.
Joe DeFuria said:
I agree. It's a great point. Hopefully it's something they have thought of. Guess we will find out as soon as they release the new build. Just hope they don't delay so much that it becomes moot(3DMark04 is possibly on the horizon).
Tommy McClain
Excellent. I've been wondering how close you'd cut your self-imposed deadline....
So, the main question remains, as others have noted, how you will notify users and reviewers which drivers FM approves to give accurate benchmark numbers? Am I to deduce from the following section that if the driver is mentioned on your website, then it's approved?
Or will you continue to link to new drivers on your frontpage, and showcase a list of approved drivers elsewhere?
So, the main question remains, as others have noted, how you will notify users and reviewers which drivers FM approves to give accurate benchmark numbers? Am I to deduce from the following section that if the driver is mentioned on your website, then it's approved?
Or will you continue to link to new drivers on your frontpage, and showcase a list of approved drivers elsewhere?
When will the new end user licence come out that states only futuremark validated drivers are to be used? because people have sine the latest version of 3d2k3 been publishing results so it can't already be out.
Because no driver yet has any reviewed comment on them in your list.
Because no driver yet has any reviewed comment on them in your list.
Ironically I sent a similar set of questions to these to both ATI and NVIDIA the day before these enforcement rules were put in place:
Neither has answered as yet.
Neither has answered as yet.
One thing I would like to know - Are the beta and development partners going to be under any kind of NDA, or are they allowed to reveal any invalid optimisations that have been found and verified to the general public? Or are FutureMark going to be the only ones allowed to notify the public of any indiscretions (if they are at all)?
Yes. 8)Hanners said:
I'm disappointed, this .pdf essentially is a lame way of saying that even though you (Futuremark) had 5 weeks to figure out which drivers are legit you didn't manage to do just that. I don't know about the others, but I sure expected more than another clarifications of operations. How about some hard facts for a change?worm[Futuremark said:
cu
incurable.
They say partners but they don't say if they notify all partners or just the one who submitted the drivers :/ I'm assuming the later.
There is supposedly gonna be another document coming in the not to distant future clarifing some of the points or so a little birdy tells me.
There is supposedly gonna be another document coming in the not to distant future clarifing some of the points or so a little birdy tells me.
Similar threads
- Locked
