Futuremark Publishes Guidelines for Driver Optimizations

[ClyssaN]

Lets hope the Q&A clarify how the guidelines will be enforced.

 

[ClyssaN]

Lets hope the Q&A clarify how the guidelines will be enforced.

 

[Entropy]

I'd like to go on record saying that I really liked what FutureMark had to say.
Very clear, short and to the point.

I'll further say that if they actually provide a quality benchmark, with a solid set of rules that are enforced, ensuring that the data produced and going into the ORB is valid, then they are providing a very good and useful product indeed.

It all hinges on the integrity of the produced data though, so the question now is obviously enforcement.

The current situation is such that nVidia isn't in a position to cause FutureMark too much damage if they take a firm stance, so they really should. And if they can establish a firm corporate policy the problems going forward should diminish automatically, because IHV cheating will be visible and embarrassing, as it well should be.

FutureMark try to supply a number of products to different buyers, but the strength of their name is intimately tied to 3DMark. If they very clearly and visibly try to keep the integrity and quality of 3DMark high, that will not only help them maintain and improve the standing of their flagship product, but will also increase the attractiveness and therefore value of the rest of their portfolio.

Lets hope they don't mess up.

Entropy

 

[Sabastian]

In its sole discretion, Futuremark may approve detection in order to fix a specified hardware error.

Gesh, so now that nvidia is back in the program they make allowances for bad hardware to compete against competitors? Come on, give me a break. What sorts of hardware problems are being masked by this and is there any penalty on the scoring?

The above reads like a loop hole ( A huge one at that.) where nvidia could simply excuse all sorts of optimizations based on Futuremarks approval of specific hardware problems.

Will the end users be made aware that their hardware is broken and that optimizations are being used to make up for the deficiencies in the hardware or no?

As Ratchet brought up will the old scores with un-approved of optimizations be removed or at least corrected?

 

[AzBat]

Joe,

Well done. I think I pretty much agree with your comments. However, I would like to make my own comments first and then follow up on your comments individually.

First of all, I'm quite surprised that they delayed their initial announcement and in the end it took another 4 days before they were published. Something tells me there was a lot of last minute discussions. I would have loved to been a fly on the wall.

All I've got to say is that come September 19, Futuremark had better be acknowledging that NVIDIA and its drivers not living up to your rules and guidelines and that their scores will not be supported. If you end-up releasing a crap set of rules and then don't have the back-bone to enforce them, then be ready to pack-up your office belongings and find another job because I don't think your company will be around much longer.

I've included my last comments here as a reminder of my position. I'm still not changing it. I'm glad some of you found the comments rather surprising for being so harsh. That was my intent as it described my feelings of Patric's asinine comments. It would be interesting to see if Patric has any more comments now that the new rules have been released.

Now onto my thoughts...

I first would like to say that I believe these rules are a step in the right direction. They have the potential to give back Futuremark their integrity and relevance to their benchmark. So in that regard I applaud them for making these rules. Are they the best? No. Are they the worst? No, not by a long shot. But like I said earlier, rules by themselves are not going to save them. Enforcement and clarification is needed.

Rule 1: Nice one. Currently I can't see how this can be misinterpreted. Meaning, if they ask for tri-linear filtering, you better give them tri-linear filtering.

Rule 2: I like the first part of the rule since it states "indirectly". However, I don't like the idea of the "but...". Will the approval process and what is approved(IHV and the detection needed) be made public? What's considered a hardware error?

Rule 3: If I understand this right, this means you can't look at the benchmark and then create clipping planes based on what was rendered on screen. Same goes for the PrtScrn issue. Bravo.

Rule 4: I like this one as well. Basically if you make an optimization in your driver, you better make sure as hell it works on other applications that use the same thing and that there is no difference in the output. Unfortunately they left a loop hole by saying "mathematically consistent". I'm sure they got some grief for the wording here. They probably used the word "correct" instead, but then had to change it because there are no correct definitions. The reference rasterizer should be correct, but then again isn't it based on some IHV's hardware? If your hardware wasn't used as the reference rasterizer, then I could see how other IHVs might have a problem with it.

All in all, I think these rules explain in enough detail of how IHVs are suppose to act. Unfortunately, I initially see some of them trying to test to see if or how Futuremark enforces the rules. That brings up my biggest question, how are Futuremark going to enforce them? Do they now have some kind of process that beta members are now required to follow? What about companies that are not members? Does Futuremark now have a process similar to WHQL? Meaning, are they are now required to send Futuremark drivers for testing before they can be approved? Is Futuremark going to make public which public driver sets are approved and which ones are not? Will they state why they weren't approved? Or are they going to do something totally different and use help from the press or beta members to help them do the policing? There are lot of questions still left that I believe need answering. I also would like to see official responses from the beta members. Does NVIDIA plan on playing by the rules?

Now, my last concern is what happens if and when drivers are found to violate these rules? Will the results just not be included in the ORB results or will there be other repercussions? Will the violations be publicly made? Also, are these rules legally binding? Will they be included in the license agreement? I don't want a IHV to violate the rules and then state they're not legally required to do so because of some kind of loop hole.

My thoughts:

Common sensical guidelines. If I know large corporations at all, the last month or two were spent dealing with disputes over specific semantic wordings. To be expected.

I would like to echo some of the follow up questions, and have suggestions:

1) To make the new guidelines clear, you should point out examples in past drivers that have broken these rules (and what optimizations DON'T break these rules). This gives us a more clear understanding of what is, and equally important, what is not a rule breaker.

Agreed.

2) Identify the most recent driver set from all contributing IHVs that to the best of Futuremark's knowledge, don't break these rules. What good are the guidelines if the users don't know which of the existing drviers are legit?

Agreed.

3) How will 3DMark "police" their guidelines? Rely on 3rd party investigations...do their own set of tests....combination of both?

Agreed. This is one of my biggest questions. I'd rather Futuremark have an official approval process that's made public and not rely on 3rd parties for the policing.

There are legitimate concerns about the "vagueness" of a couple of the rules. (Understandably, the rules can't be too specific, because then they are at risk of being bypassed with loopholes.)

What would alleviate some concern about rules being bent improperly, (for example with rule number 2) would be for Futuremark to agree to make public those cases where it approves application detection. The public needs to know about any "hardware error" that relies on application detection to be addressed. If for no other reason that this means similar application detection would be needed for every other game that uses similar techniques....and if one card relies on app detection and one doesn't, that means there is additional risk invovled in purchasing a card that needs application detection.

I would love for Futuremark to make public the approved detections(if any), but I'm not sure if IHVs would allow Futuremark to announce any of them. Announcing the "hardware errors" could potentially scare away customers that see the hardware as being defective. I could see the potential for some hardware having glitches that requires some special casing in the drivers, but not being so bad that the hardware is useless for the tasks it was purchased for. But yes, it would be nice to keep the customer informed so they know what they're getting into.

In closing, I would like to say that these new rules show me that Futuremark is finally seeing that they need to protect 3DMark from cheating. Unfortunately they don't go into any details on how they expect to enforce these new rules. Nor do they say if any IHVs/drivers violate currently violate those rules. I believe once we hear more details on those issues we will have a better idea on whether 3DMark and Futuremark have regained the trust and integrity that is needed to be the industry leading benchmark. You can be sure I'll voice my opinion once they do.

Tommy McClain

 

[nyt]

Will IHVs caught cheating be legally forced/bound by agreement to remove invalid scores from their PR? After all, it's the credibility of 3dmark which is exposed thru the results and advertising.

 

[OpenGL guy]

Rule 4: I like this one as well. Basically if you make an optimization in your driver, you better make sure as hell it works on other applications that use the same thing and that there is no difference in the output. Unfortunately they left a loop hole by saying "mathematically consistent". I'm sure they got some grief for the wording here. They probably used the word "correct" instead, but then had to change it because there are no correct definitions. The reference rasterizer should be correct, but then again isn't it based on some IHV's hardware? If your hardware wasn't used as the reference rasterizer, then I could see how other IHVs might have a problem with it.

No one is going to perfectly match the refrast. The reason being that the refrast runs on a CPU with likely much higher internal precision than any consumer graphics hardware. Also, just because there are minor differences in LSBs, doesn't mean that your result is not correct.

There are lots of other types of precision that come into play in your result. Depth precision, vertex position, texture coordinates, texture sampling/filtering, subpixel precision, etc. It's highly unlikely that any two algorithms will give exactly the same results when you take all of these factors into account.

This doesn't mean that you can't determine a right answer from a wrong answer, however. You just need to account for the fact that there may be more than one correct result.

Edit: Fix typo.

 

[bloodbob]

Umm so the current drivers for nvidia are already out unless 3Dmark uses Bi-linear filtering? Because it doesn't do true tri-linear filtering.

 

[Borsti]

I think it´s a good start.

But what happens if one manufacturer violates those rules? Only a ban of this driver in the ORB? This won´t help with drivers specially created to get OEM wins.

What I would like to see is something like this (just a idea): No optimization, detection etc. allowed at all. If one manufacturer wants to implement certain optimizations, they should first talk to FM. FM then can descide if they are valid and offer a patch. All optimizations should be made public to all beta members.

I also told something similar to the AM3 guys. Watermark the shaders used in the game and if the app detects that another shader is used (may be possible via their special features) then no score is given because of invalid drivers.

The problem I see for the current rules are that they won´t help the press to detect invalid press drivers and they won´t help the OEM´s. You may get that information when it´s to late with the current rules.

Lars

 

[OpenGL guy]

What I would like to see is something like this (just a idea): No optimization, detection etc. allowed at all. If one manufacturer wants to implement certain optimizations, they should first talk to FM. FM then can descide if they are valid and offer a patch. All optimizations should be made public to all beta members.

I don't believe this will work. The reason is that what helps one vendor may harm another. Also, it's not always possible to fully optimize your software for the hardware within the limits of the API.

 

[Bambers]

On point 4.

How exactly are you defining mathematically consistent?

Will there be a limit on how far off in total the image is or how far off one pixel can be or both or something else?

 

[madshi]

I like the rules. Like a lot of other people already said, the biggest question is now how FutureMark will enforce the rules. And how it will react if an IHV doesn't follow the rules.

I'd like the following solution: Before doing any benchmarks 3DMark should go online and ask an online database whether the drivers are alright. If they are known to contain invalid optimizations, the benchmark shouldn't be executed at all. Instead a big warning should be there, saying which invalid optimizations were found in that specific driver.

That would mean that all IHVs would have to give their drivers to FutureMark for validation.

I know, this would be quite a lot of work for FutureMark. But they could take money for the validation work. And this way you might be able to get some of the lost community respect back.

 

[Borsti]

What I would like to see is something like this (just a idea): No optimization, detection etc. allowed at all. If one manufacturer wants to implement certain optimizations, they should first talk to FM. FM then can descide if they are valid and offer a patch. All optimizations should be made public to all beta members.

I don't believe this will work. The reason is that what helps one vendor may harm another. Also, it's not always possible to fully optimize your software for the hardware within the limits of the API.

Well... if the app contains codepaths for the different vendors/cards an optimization would not harm the others. Since all code would be public to the beta members, everybody will know what those optimizations are and cry foul if something´s fishy (regarding shader replacement etc).

It´s just an idea. But in the end those rules won´t help me when I get a new driver from an IHV.

Lars

 

[Simon F]

]I'll post this here too as this topic seems to be discussed in two threads.

If you guys have any questions concerning the ground rules, please post them in this thread, or over at our boards in this thread. We will collect the most frequently asked questions and publish a QA within the next few days. Thanks!

Sorry worm,
I couldn't be bothered registering at yet another forum so you can read my question/comment here:

http://www.beyond3d.com/forum/viewtopic.php?p=170823#170823


Feel free to copy and paste into FutureMark's forums.

 

[Dio]

I also told something similar to the AM3 guys. Watermark the shaders used in the game and if the app detects that another shader is used (may be possible via their special features) then no score is given because of invalid drivers.

The problem with this is threefold:
- I have no idea how you would accomplish this watermarking, given that you would be relying on the driver to give you back any information about the rendered result
- shaders will change as compilers improve, in the same way that the executable code produced by MSVC 6.0 SP5PP5 isn't necessarily the same as that produced by MSVC 6.0 SP4PP4. For example, there may be tiny differences introduced because a new compiler changes order-of-evaluation
- just because 'something changes' doesn't imply 'the results are worse'. It's perfectly forseeable that a driver could fix bugs, or improve image quality, but then be flagged up for cheating because 'something has changed'

 

[nyt]

What about only accepting known good drivers, instead of rejecting cheating ones? It's FM responsability to validate the results, since we can't trust some IHVs anyway. A simple online automated query could update the known good driver list at each launch, and that's it.
IMHO, what people expect from FM is to show R3xx = twice as fast as NV3x

 

[StealthHawk]

It is prohibited to detect 3DMark directly or indirectly. In its sole discretion, Futuremark may approve detection in order to fix a specified hardware error

Read "Loophole for Nvidia. Please take your lawyers and go home.. btw.. thanks for the money."

Yes, this sounds rather ominous. It seems much too vague. Are we talking about a hardware error that doesn't allow the proper image to be rendered(like the NV30 fog bug which has since been worked around in software). Hell, NV3x's register use problems might fall under a "hardware error" for all we know :?

Rule 4: I like this one as well. Basically if you make an optimization in your driver, you better make sure as hell it works on other applications that use the same thing and that there is no difference in the output. Unfortunately they left a loop hole by saying "mathematically consistent". I'm sure they got some grief for the wording here. They probably used the word "correct" instead, but then had to change it because there are no correct definitions. The reference rasterizer should be correct, but then again isn't it based on some IHV's hardware? If your hardware wasn't used as the reference rasterizer, then I could see how other IHVs might have a problem with it.

Are you implying that some actual IHV's hardware was used as the basis for the reference rasterizer? We know R3xx cannot be, since the reference rasterizer uses FP32. But even NVIDIA employees have said that the reference rasterizer will not produce the same image as NV3x. So whether or not the reference rasterizer "favors" one IHV over another, no IHV will have hardware that will perfectly match the rasterizer. The "correct" image should be the image produced by the hardware under normal conditions, bugs not withstanding.

 

[Borsti]

I also told something similar to the AM3 guys. Watermark the shaders used in the game and if the app detects that another shader is used (may be possible via their special features) then no score is given because of invalid drivers.

The problem with this is threefold:
- I have no idea how you would accomplish this watermarking, given that you would be relying on the driver to give you back any information about the rendered result
- shaders will change as compilers improve, in the same way that the executable code produced by MSVC 6.0 SP5PP5 isn't necessarily the same as that produced by MSVC 6.0 SP4PP4. For example, there may be tiny differences introduced because a new compiler changes order-of-evaluation
- just because 'something changes' doesn't imply 'the results are worse'. It's perfectly forseeable that a driver could fix bugs, or improve image quality, but then be flagged up for cheating because 'something has changed'

Hm... AM3 has the option to show the shaders used by the driver:

Massive: "Due to this fact Due to this fact it is necessary to detect the number of pixel which finally come through the pipeline and contribute to the framebuffer result. This is difficult, because in nowadays pixelpipelines many pixel are rejected at different stages of the pipeline by z/R-tests. We invented a new testing method which exactly counts the number of pixel contributing to the framebuffer. The special thing about thist is, that it does not change any etting of the pixelpipeline, so that the counting pass delivers exactly the same number than the original rendering passes."

Or use an application that´s between the driver and the application to detect the shaders... tools like moreBENCH LX (http://www.more3d.com/) use such a technology to count triangles and FPS.

Lars

 

[Dave Baumann]

Lars,

that shows the shaders the application requests the driver runs - this can be entirely separates from the sahder code the driver actually runs. In the application will submit some assembly to the driver and as far as the application is concerned that is what is run, however the driver may entirely replace that sahder code with some completely different shader code or optimise the code to do something the same, but faster - the application has no way of understanding that either of these scenario's have actually occured, since its job stops once its submitted the inital assembly it wants to run.

 

[Dio]

Only with video capture on VGA/DVI can you see what's actually been rendered - and it would need to be DVI to do exact-bit-compares.