[ClyssaN]

http://www.futuremark.com/pressroom/...leases/?092303
reading ...

edit: MightyPez post it here :
http://www.beyond3d.com/forum/viewto...8016&start=100

I think it deserves a new topic, you mods decide.

edit: link fixed, tks natoma

[digitalwanderer]

I like what they're saying, but I'll hold off on celebrating until I see how well they enforce it.

[Natoma]

That link doesn't work. This one does.

http://www.futuremark.com/pressroom/...leases/?092303

[Nick[FM]]

I'll post this here too as this topic seems to be discussed in two threads.

If you guys have any questions concerning the ground rules, please post them in this thread, or over at our boards in this thread. We will collect the most frequently asked questions and publish a QA within the next few days. Thanks!

[ClyssaN]

Originally Posted by digitalwanderer

I like what they're saying, but I'll hold off on celebrating until I see how well they enforce it.

I agree.

[digitalwanderer]

Originally Posted by worm[Futuremark

]I'll post this here too as this topic seems to be discussed in two threads.

If you guys have any questions concerning the ground rules, please post them in this thread, or over at our boards in this thread. We will collect the most frequently asked questions and publish a QA within the next few days. Thanks!

How does Futuremark plan to enforce it's new guidelines when an IHV is exposed to be cheating somehow in their drivers?

[Mark]

Fairly common-sense, those rules. Took you guys a month to come up with that? Regardless, let's see if you guys actually enforces them this time.

Since they obviously break all of these "new" rules, when can we expect the GeForceFX scores to get removed from the ORB and Hall of Fame lists?[/b]

[xGL]

I don't like this as it can be interpreted in many, many ways :

In its sole discretion, Futuremark may approve detection in order to fix a specified hardware error.

[ClyssaN]

Originally Posted by xGL

I don't like this as it can be interpreted in many, many ways :

I dont think nv30/35 can "escape" using that quote...

[Nautis]

Whats your policy on bugs introduced in a driver? If a bug in some way enhances the preformance of 3dmark is there a timeframe which you'll give a company to fix that bug before the results are pulled?

And to expand on DWs question: In what way and at what time do you plan to inform the public of any optimizations or performance enhancing bugs that may occour from this point?

-

I have to say I am also concerned with the vagueness of these guidelines (Though I cant say I didnt expect it.)

For anyone who may know: Did futuremark consult the public in any way before making these guidelines or did they only consult their beta partners?

BTW Dave whats the status of B3D in the beta program and whats the official stance on this particular case of pr?

[digitalwanderer]

Another couple of 'em I posted up at FM's thread about it:

Which current detonator drivers are considered "legal" for 3dm2k3?

Will you be reviewing driver sets before allowing them to post scores, or will you be investigating drivers only when there is a reported problem/cheat?

[ClyssaN]

Originally Posted by digitalwanderer

Will you be reviewing driver sets before allowing them to post scores, or will you be investigating drivers only when there is a reported problem/cheat?

That is a good question.

I how fast FM will act is another one.

[CorwinB]

Those are nice rules of thumb, Worm. Of course, we will see when the time to enforce (*cough* Cheatonators 51.75 *cough*) comes...

Those rules have been reviewed and accepted by *all* your beta partners, correct ?

BTW, any clarification on what drivers are allowed for 3DMark ? What is FutureMark's official stance on websites/publications using beta or "special PuRpose" drivers for benchmarking with 3DMark ?

[Hellbinder]

It is prohibited to detect 3DMark directly or indirectly. In its sole discretion, Futuremark may approve detection in order to fix a specified hardware error

Read "Loophole for Nvidia. Please take your lawyers and go home.. btw.. thanks for the money."

[digitalwanderer]

Originally Posted by Hellbinder

Read "Loophole for Nvidia. Please take your lawyers and go home.. btw.. thanks for the money."

I don't think "sucks butt actually running DX9" counts as a specified hardware error....

....at least I sure hope it doesn't.

[crystalcube]

Ok so that guideline are here but how will they be enforced?
so thats still behind closed doors ... ?
maybe give a RED card if anyone doesn't follows these guidelines

[Joe DeFuria]

My thoughts:

Common sensical guidelines. If I know large corporations at all, the last month or two were spent dealing with disputes over specific semantic wordings. To be expected.

I would like to echo some of the follow up questions, and have suggestions:

1) To make the new guidelines clear, you should point out examples in past drivers that have broken these rules (and what optimizations DON'T break these rules). This gives us a more clear understanding of what is, and equally important, what is not a rule breaker.

2) Identify the most recent driver set from all contributing IHVs that to the best of Futuremark's knowledge, don't break these rules. What good are the guidelines if the users don't know which of the existing drviers are legit?

3) How will 3DMark "police" their guidelines? Rely on 3rd party investigations...do their own set of tests....combination of both?

There are legitimate concerns about the "vagueness" of a couple of the rules. (Understandably, the rules can't be too specific, because then they are at risk of being bypassed with loopholes.)

What would alleviate some concern about rules being bent improperly, (for example with rule number 2) would be for Futuremark to agree to make public those cases where it approves application detection. The public needs to know about any "hardware error" that relies on application detection to be addressed. If for no other reason that this means similar application detection would be needed for every other game that uses similar techniques....and if one card relies on app detection and one doesn't, that means there is additional risk invovled in purchasing a card that needs application detection.

[nelg]

Since Worm asked I will copy my comment from the other thread....

I would like to see a form of punishment in place whereby if a IHV is found to be cheating its drivers containing the cheats would be disqualified. Furthermore the next driver release from that company should automatically have its score reduced by 10%-20% for period of time. Consider it a form of probation. This would give Futuremark and others time to investigate and would prevent IHV’s from continually releasing drivers for the purpose of circumventing the guidelines.

[AzBat]

Sorry for the post. Accidently hit "Submit" instead of "Preview". I tried for like 4 hours after I posted it to edit the message, but it kept timing out. "Preview" and posting new messages worked fine, but for some odd reason "Edit" wouldn't. Anyway, looks like its working again.

Tommy McClain

[digitalwanderer]

Originally Posted by nelg

Since Worm asked I will copy my comment from the other thread....

I would like to see a form of punishment in place whereby if a IHV is found to be cheating its drivers containing the cheats would be disqualified. Furthermore the next driver release from that company should automatically have its score reduced by 10%-20% for period of time. Consider it a form of probation. This would give Futuremark and others time to investigate and would prevent IHV’s from continually releasing drivers for the purpose of circumventing the guidelines.

I still think that publically flogging the offending IHV's CEO would be a hell of a lot more effective and fun....

[ClyssaN]

Lets hope the Q&A clarify how the guidelines will be enforced.

[ClyssaN]

Lets hope the Q&A clarify how the guidelines will be enforced.

[Entropy]

I'd like to go on record saying that I really liked what FutureMark had to say.
Very clear, short and to the point.

I'll further say that if they actually provide a quality benchmark, with a solid set of rules that are enforced, ensuring that the data produced and going into the ORB is valid, then they are providing a very good and useful product indeed.

It all hinges on the integrity of the produced data though, so the question now is obviously enforcement.

The current situation is such that nVidia isn't in a position to cause FutureMark too much damage if they take a firm stance, so they really should. And if they can establish a firm corporate policy the problems going forward should diminish automatically, because IHV cheating will be visible and embarrassing, as it well should be.

FutureMark try to supply a number of products to different buyers, but the strength of their name is intimately tied to 3DMark. If they very clearly and visibly try to keep the integrity and quality of 3DMark high, that will not only help them maintain and improve the standing of their flagship product, but will also increase the attractiveness and therefore value of the rest of their portfolio.

Lets hope they don't mess up.

Entropy

[Sabastian]

In its sole discretion, Futuremark may approve detection in order to fix a specified hardware error.

Gesh, so now that nvidia is back in the program they make allowances for bad hardware to compete against competitors? Come on, give me a break. What sorts of hardware problems are being masked by this and is there any penalty on the scoring?

The above reads like a loop hole ( A huge one at that.) where nvidia could simply excuse all sorts of optimizations based on Futuremarks approval of specific hardware problems.

Will the end users be made aware that their hardware is broken and that optimizations are being used to make up for the deficiencies in the hardware or no?

As Ratchet brought up will the old scores with un-approved of optimizations be removed or at least corrected?

[AzBat]

Joe,

Well done. I think I pretty much agree with your comments. However, I would like to make my own comments first and then follow up on your comments individually.

First of all, I'm quite surprised that they delayed their initial announcement and in the end it took another 4 days before they were published. Something tells me there was a lot of last minute discussions. I would have loved to been a fly on the wall.

Originally Posted by AzBat

All I've got to say is that come September 19, Futuremark had better be acknowledging that NVIDIA and its drivers not living up to your rules and guidelines and that their scores will not be supported. If you end-up releasing a crap set of rules and then don't have the back-bone to enforce them, then be ready to pack-up your office belongings and find another job because I don't think your company will be around much longer.

I've included my last comments here as a reminder of my position. I'm still not changing it. I'm glad some of you found the comments rather surprising for being so harsh. That was my intent as it described my feelings of Patric's asinine comments. It would be interesting to see if Patric has any more comments now that the new rules have been released.

Now onto my thoughts...

I first would like to say that I believe these rules are a step in the right direction. They have the potential to give back Futuremark their integrity and relevance to their benchmark. So in that regard I applaud them for making these rules. Are they the best? No. Are they the worst? No, not by a long shot. But like I said earlier, rules by themselves are not going to save them. Enforcement and clarification is needed.

Rule 1: Nice one. Currently I can't see how this can be misinterpreted. Meaning, if they ask for tri-linear filtering, you better give them tri-linear filtering.

Rule 2: I like the first part of the rule since it states "indirectly". However, I don't like the idea of the "but...". Will the approval process and what is approved(IHV and the detection needed) be made public? What's considered a hardware error?

Rule 3: If I understand this right, this means you can't look at the benchmark and then create clipping planes based on what was rendered on screen. Same goes for the PrtScrn issue. Bravo.

Rule 4: I like this one as well. Basically if you make an optimization in your driver, you better make sure as hell it works on other applications that use the same thing and that there is no difference in the output. Unfortunately they left a loop hole by saying "mathematically consistent". I'm sure they got some grief for the wording here. They probably used the word "correct" instead, but then had to change it because there are no correct definitions. The reference rasterizer should be correct, but then again isn't it based on some IHV's hardware? If your hardware wasn't used as the reference rasterizer, then I could see how other IHVs might have a problem with it.

All in all, I think these rules explain in enough detail of how IHVs are suppose to act. Unfortunately, I initially see some of them trying to test to see if or how Futuremark enforces the rules. That brings up my biggest question, how are Futuremark going to enforce them? Do they now have some kind of process that beta members are now required to follow? What about companies that are not members? Does Futuremark now have a process similar to WHQL? Meaning, are they are now required to send Futuremark drivers for testing before they can be approved? Is Futuremark going to make public which public driver sets are approved and which ones are not? Will they state why they weren't approved? Or are they going to do something totally different and use help from the press or beta members to help them do the policing? There are lot of questions still left that I believe need answering. I also would like to see official responses from the beta members. Does NVIDIA plan on playing by the rules?

Now, my last concern is what happens if and when drivers are found to violate these rules? Will the results just not be included in the ORB results or will there be other repercussions? Will the violations be publicly made? Also, are these rules legally binding? Will they be included in the license agreement? I don't want a IHV to violate the rules and then state they're not legally required to do so because of some kind of loop hole.

Originally Posted by Joe DeFuria

My thoughts:

Common sensical guidelines. If I know large corporations at all, the last month or two were spent dealing with disputes over specific semantic wordings. To be expected.

I would like to echo some of the follow up questions, and have suggestions:

1) To make the new guidelines clear, you should point out examples in past drivers that have broken these rules (and what optimizations DON'T break these rules). This gives us a more clear understanding of what is, and equally important, what is not a rule breaker.

Agreed.

Originally Posted by Joe DeFuria

2) Identify the most recent driver set from all contributing IHVs that to the best of Futuremark's knowledge, don't break these rules. What good are the guidelines if the users don't know which of the existing drviers are legit?

Agreed.

Originally Posted by Joe DeFuria

3) How will 3DMark "police" their guidelines? Rely on 3rd party investigations...do their own set of tests....combination of both?

Agreed. This is one of my biggest questions. I'd rather Futuremark have an official approval process that's made public and not rely on 3rd parties for the policing.

Originally Posted by Joe DeFuria

There are legitimate concerns about the "vagueness" of a couple of the rules. (Understandably, the rules can't be too specific, because then they are at risk of being bypassed with loopholes.)

What would alleviate some concern about rules being bent improperly, (for example with rule number 2) would be for Futuremark to agree to make public those cases where it approves application detection. The public needs to know about any "hardware error" that relies on application detection to be addressed. If for no other reason that this means similar application detection would be needed for every other game that uses similar techniques....and if one card relies on app detection and one doesn't, that means there is additional risk invovled in purchasing a card that needs application detection.

I would love for Futuremark to make public the approved detections(if any), but I'm not sure if IHVs would allow Futuremark to announce any of them. Announcing the "hardware errors" could potentially scare away customers that see the hardware as being defective. I could see the potential for some hardware having glitches that requires some special casing in the drivers, but not being so bad that the hardware is useless for the tasks it was purchased for. But yes, it would be nice to keep the customer informed so they know what they're getting into.

In closing, I would like to say that these new rules show me that Futuremark is finally seeing that they need to protect 3DMark from cheating. Unfortunately they don't go into any details on how they expect to enforce these new rules. Nor do they say if any IHVs/drivers violate currently violate those rules. I believe once we hear more details on those issues we will have a better idea on whether 3DMark and Futuremark have regained the trust and integrity that is needed to be the industry leading benchmark. You can be sure I'll voice my opinion once they do.

Tommy McClain