Process to Enforce 3DMark® Optimization Guidelines

.pdf said:
Inform BDP Members

Futuremark informs BDP members about potential issues found in driver it has reviewed. Futuremark also informs BDP members if the suspected issues did not exist and the driver was found to be fulfilling the guidelines.

If I understand this right, Futuremark informs BDP members of mentioned issues before contacting the manufacturer about it?
Doesn't that raise potential problems?
What if a "legitimate" bug finds its way into the drivers that reveals a confidential aspect of HW/driver technology to competing companies?

OK, I'm down to calling bugs legitimate. what next?
 
vb said:
If I understand this right, Futuremark informs BDP members of mentioned issues before contacting the manufacturer about it?

??

It doesn't seem like it's BEFORE contacting the manufacturer at all:

Futuremark said:
If Futuremark suspects that a driver breaches the guidelines, it will contact the manufacturer immediately and demand an explanation or corrective action within one week.

The manufacturer is contacted immediately.

Though it's not clear to me when exactly the BDP members are contacted in the process. At the same time that the manufacturer is? Or after the 1 week deadline for the manufacturer's response?

Based on the block diagram, it looks like the BDP members and the manufacturer are informed about potential issues at the same time. This is reasonable, as long as the BDP members are also kept in the immediate loop on progress made to the resolution on the issue. (Which appears to be the case. That is, when the manufacturer gives the response to FM, FM passes that information on to the BDP.

Informing BDP and the manufacturer at the same time puts some pressure on the manufacturer to respond more quickly with some response, which is a good thing.

The obvious question that I was hoping would be answered at this time is: what's the sate of the current WHQL drivers from all vendors and their products?

Which raises a related question: drivers behanve differently depending on the product using them. It's entirely feasible, if not likely, that one piece of hardware and driver X may "pass" FutureMark tests, while a different piece of hardware with the SAME driver could fail. Is FutureMark preparing the patch / driver "review" database with this in mind?

In other words, it's not enough to label a driver version as "reviewed" or not. It has to be a combination of driver / chip-level hardware.
 
This is definately a step in the right direction, and if I may say- has been a long time coming.

Discounting selective discretion with what are deemed "valid drivers" and to what fuzzy degree they adhere to given benchmark driver guidelines, this whole thing now boils down to two issues-

1) ORB/FutureMark website results (which this PDF outlines, but fails to mention if such changes will be retroactive.. i.e. does it only effect newly attempted published scores or will previous scores on drivers deemed improper be purged from the database?)

2) Website/3rd party usage of the benchmark when using 3DMark scores for product comparisons in the context of website reviews.

Issue #2 is not compensated nor handled by anything I've seen so far, and is actually the primary problem or concern. I'm not sure what market research has been performed, but I don't see IHV hardware purchase decisions/comparisons having much reliance on ORB score results. ORB results are generally from people with the already-purchased hardware, so I can easily see any IHV being satisfied with a simple repercussion of "you can't publish cheating scores"... just as long as Website or Magazine A,B and C will continue to publish graphs with the overly inflated and inaccurate score comparisons to sell units.

The true solution is to advance the product to fail to deliver scores on uncertified drivers. A simple WHQL certification check would be the best first pass, and with some form of downloadable "certified" driver system (much like Antivirus software definitions) to allow the product to even produce a score. Obviously such a change would be a long term goal, but such discussion/dialogue on such a future would more clearly show the motives of FutureMark and the interest of the accuracy of their products.
 
Blimey - has it been a month already? I completely fortgot this was due today.

Worm - nice to see you guys make it out on time with this ;)

I like the conditions you have presented and look forward to seeing futuremark evolve with its software.
 
Compliments to FutureMark for putting it out in a timely manner and my thanks to Worm for coming around and telling us 'bout it. :)

I still haven't actually READ it yet though... (I hate .pdfs! Where is the .html version? ;) )
 
Joe DeFuria said:
vb said:
If I understand this right, Futuremark informs BDP members of mentioned issues before contacting the manufacturer about it?

??

It doesn't seem like it's BEFORE contacting the manufacturer at all:

My mistake. I meant before receiving a response about it from the manufacturer.

Which i still think is not quite fair
 
vb said:
If I understand this right, Futuremark informs BDP members of mentioned issues before contacting the manufacturer about it?

Hmm, I must obvioulsy read wörms .pdf again :oops:
The idea is to first contact the driver manufacturer.
 
worm[Futuremark said:
]We just posted a new pdf about the enforcement of the optimization guidelines:

http://www.futuremark.com/companyinfo/Enforcement_Process.pdf

Worm,

Thanks for the link and the heads up! I just checked the PDF and have to say that I don't have much criticisms, if any, so far of your actual process. Good job. Unfortunately, there is one thing I have a problem with...

Joe DeFuria said:
The obvious question that I was hoping would be answered at this time is: what's the sate of the current WHQL drivers from all vendors and their products?

Agreed. I'm a little disappointed that the Oct. 31st date wasn't adhere to for notifying us which drivers are certified for use with 3DMark. In my last post on this topic...

http://www.beyond3d.com/forum/viewtopic.php?t=8157&start=40

I stated...

AzBat said:
Futuremark definitely made some progress with their QA. I'm very pleased with them. I wasn't too surprised to hear that they haven't approved of any current drivers. So in effect, all IHVs are all on the same page until Oct. 31st. The 35 day wait is understandable and agreeable. Like others have said provided Futuremark doesn't push it back farther. Using their previous track record I can see it happening though. If they do, then I'm not going to be very pleased at all.

Looks like they haven't pushed the date back per se, but they did say they would release a new build of 3DMark. Unfortunately, they give no timeline for the release of the new build or the list of drivers that pass the certification. I know things take time, but the guidelines specifically said they would "begin full-fledged enforcing at latest on October 31st.". Nothing in the PDF today tells me if they're actually doing that.

Joe DeFuria said:
Which raises a related question: drivers behanve differently depending on the product using them. It's entirely feasible, if not likely, that one piece of hardware and driver X may "pass" FutureMark tests, while a different piece of hardware with the SAME driver could fail. Is FutureMark preparing the patch / driver "review" database with this in mind?

In other words, it's not enough to label a driver version as "reviewed" or not. It has to be a combination of driver / chip-level hardware.

I agree. It's a great point. Hopefully it's something they have thought of. Guess we will find out as soon as they release the new build. Just hope they don't delay so much that it becomes moot(3DMark04 is possibly on the horizon).

Tommy McClain
 
Excellent. I've been wondering how close you'd cut your self-imposed deadline.... :)

So, the main question remains, as others have noted, how you will notify users and reviewers which drivers FM approves to give accurate benchmark numbers? Am I to deduce from the following section that if the driver is mentioned on your website, then it's approved?

There are two possible outcomes of the above:
1) Futuremark’s suspicion was proved to be wrong and in fact the driver met the Optimization
Guidelines. In this case, Futuremark informs its BDP members of the results and publishes
information of the driver in its web pages with status: ‘Reviewed’; or
2) The driver did not fulfill the Optimization Guidelines. In this case, Futuremark informs its BDP
members of the results and does not publish information of the driver on its web site

Or will you continue to link to new drivers on your frontpage, and showcase a list of approved drivers elsewhere?
 
When will the new end user licence come out that states only futuremark validated drivers are to be used? because people have sine the latest version of 3d2k3 been publishing results so it can't already be out.

Because no driver yet has any reviewed comment on them in your list.
 
Ironically I sent a similar set of questions to these to both ATI and NVIDIA the day before these enforcement rules were put in place:

Since rejoining Futuremarks Beta program, does NVIDIA agree to the rules and guidelines set by Futuremark?

Futuremark released their 3DMark03 optimisation guidelines a few weeks back - did does NVIDIA have a hand in shaping these guidelines and fully approve of the final outcome?

Should Futuremark decide to enforce these guidelines by some method, would you be happy that they do so?

Are you confident that NVIDIA's publicly available drivers will pass the guidelines set forth by Futuremark?

Neither has answered as yet.
 
One thing I would like to know - Are the beta and development partners going to be under any kind of NDA, or are they allowed to reveal any invalid optimisations that have been found and verified to the general public? Or are FutureMark going to be the only ones allowed to notify the public of any indiscretions (if they are at all)?
 
Hanners said:
One thing I would like to know - Are the beta and development partners going to be under any kind of NDA, or are they allowed to reveal any invalid optimisations that have been found and verified to the general public? Or are FutureMark going to be the only ones allowed to notify the public of any indiscretions (if they are at all)?
Yes. 8)
 
worm[Futuremark said:
]We just posted a new pdf about the enforcement of the optimization guidelines:

http://www.futuremark.com/companyinfo/Enforcement_Process.pdf
I'm disappointed, this .pdf essentially is a lame way of saying that even though you (Futuremark) had 5 weeks to figure out which drivers are legit you didn't manage to do just that. I don't know about the others, but I sure expected more than another clarifications of operations. How about some hard facts for a change?

cu

incurable. :(
 
They say partners but they don't say if they notify all partners or just the one who submitted the drivers :/ I'm assuming the later.


There is supposedly gonna be another document coming in the not to distant future clarifing some of the points or so a little birdy tells me.
 
Guidelines are nice, but if they're never used for more than window dressing, then they're not worth an awful lot.

This is a truly dreadful reaction time from Futuremark. Let's hope they get a move on.
 
Back
Top