Optimization Guidelines QA

[Nick[FM]]

We just posted the QA regarding our Optimization Guidelines, and it can be viewed here (pdf).

 

[Entropy]

]We just posted the QA regarding our Optimization Guidelines, and it can be viewed here (pdf).

Without picking nits about specific wordings, the QA seems both clear and reasonable.

The remaining bothersome issue that I see is that all checking, validation, decisions on what is OK or not, is internal to FutureMark. This is an uncomfortable situation with a benchmark, as it means that it is wide open to corruption, with little possibility for public scrutiny. Still, although avoidable to some degree, this is pretty much inherent with commercial products.

Putting healthy scepticism to the side for the moment, and assuming that FutureMark actually mean what they say, it would seem they fully understand what is needed if they are to maintain their position as a prime performance gauge reference.

We've seen the words. They look pretty good.
It will be interesting to see what actually happens.

Entropy

 

[Joe DeFuria]

After my first pass reading through it:

Just a few questions:

Q: Have you been discussing the new ground rules with all members in the benchmark development program, and have they all approved them?
A: We have worked with all members in drafting these guidelines and we have solicited feedback from them throughout the process. We have gotten very strong agreement to these rules from an overwhelming majority of the members.

So, um, care to mention names of those in the "minority?"

We presume that manufacturers will want to make sure the validity of their drivers with us before they ship those to reviewers.

I think that's an optimisitc presumption. You may want to include in your terms of use, that those publishing scores (in PR releases, reviews, etc.) on non FutureMark approved drivers, must indicate such.

 

[Bjorn]

The resulting image of the optimized rendering must be as close as possible for that hardware to the rendering of the reference rasterizer. Also, the calculations performed in the
rendering process must be mathematically equivalent to those performed by the reference rasterizer.

Does this mean always FP 32 on the NV3x ? (since AFAIK, the reference rasterizer uses FP32 or higher)

Or is FP24 enough and in that case, can a mix of FP16 and FP 32 count as FP 24, assuming that the IQ are the same ?

And if that's the case, won't it be rather hard to evaulate if it's as close to FP24 as it need to be ? I mean, it would be rather easy if you just specified similar IQ when not zooming in 3-4X but we're talking about " mathematically equivalent to those performed by the reference rasterizer" here.

 

[Nick[FM]]

Without picking nits about specific wordings, the QA seems both clear and reasonable.

Good to hear! We tried to make it as "readable" as possible so that everyone understands even the more technical bits. It was kind of difficult to get all questions answered.

The remaining bothersome issue that I see is that all checking, validation, decisions on what is OK or not, is internal to FutureMark. This is an uncomfortable situation with a benchmark, as it means that it is wide open to corruption, with little possibility for public scrutiny. Still, although avoidable to some degree, this is pretty much inherent with commercial products.

You can rest assured that even if the whole process will be internal, we want to make sure that any 3DMark results posted in public publications are correct. We will post a detailed description of the whole process as soon as possible. It should make the whole process a bit more easy to understand and clear.

I think that's an optimisitc presumption. You may want to include in your terms of use, that those publishing scores (in PR releases, reviews, etc.) on non FutureMark approved drivers, must indicate such.

Yes. We are striving for that no public presentation, no review etc. is published with non 3DMark approved drivers. It will require a lot of work from us, but we believe it is worth the effort. I also think that manufacturers want to co-operate on this in sending in their new drivers to be "ok'd" before they publish any 3DMark results in public, or send them to reviewers. It is also important that reviewers let us know if they have got new hardware and new drivers, but no confirmation about the usage of 3DMark. I think this is a very good thing, and we hope that the media takes the initiative to contact us if anything is unclear.

 

[jb]

worm,

thanks for sharing and the work. I agre its sounds and looks great. Now I hope you can carry it out. And please forgive me for being a bit shy here as sometimes in the past I have been burned in this mannor (saying someone will do something and yet they never do). So I hope you can understand...you know once..once bitten twice shy..

 

[Joe DeFuria]

Does this mean always FP 32 on the NV3x ? (since AFAIK, the reference rasterizer uses FP32 or higher)

I'm not sure if the reference rasterizer supports lower precision when the PP hint is used, or if it just renders FP32 anyway.

However, from the rules as a whole, I take it that FP32 will be required (on nVidia hardware)...only when "full precision" is requested by 3DMark.

nVidia will not be allowed to force 16 bit if FutureMark doesn't request it.

 

[Nick[FM]]

Bjorn,

2 small clips from the guidelines:

1. It is prohibited to change the rendering quality level that is requested by 3DMark.

and

2. It is prohibited to detect 3DMark directly or indirectly.

Those should answer your question. Or did I misunderstand your question (as usual) ?

 

[Nick[FM]]

worm,

thanks for sharing and the work. I agre its sounds and looks great. Now I hope you can carry it out. And please forgive me for being a bit shy here as sometimes in the past I have been burned in this mannor (saying someone will do something and yet they never do). So I hope you can understand...you know once..once bitten twice shy..

Good to hear that the whole package sounds and looks great! We will do our very best that all the work we have been doing the whole summer (and part of this fall) will happen. It hasn't been like a walk in the park, but I think the results are good, and the feedback we have got has been very positive!

 

[Joe DeFuria]

]Yes. We are striving for that no public presentation, no review etc. is published with non 3DMark approved drivers. It will require a lot of work from us, but we believe it is worth the effort.

If you can pull that off....that would indeed be something. That is above and beyond my expectations actually. Glad to see you pursuing it! And yes, it will mean a lot of work for you, and it's appreciated.

I also think that manufacturers want to co-operate on this in sending in their new drivers to be "ok'd" before they publish any 3DMark results in public, or send them to reviewers.

I would have to disagree with that...or that (some) IHVs are paying little more than lip service to you on this. I assume there is going to be some sort of "turn around" time to get FMQL (tm) certified. They are already taking the time and effort to get WHQL certification, which is probably constraining to them. FMQL will likely be seen as another constraint to them...especially in times when they want to get a "special" *cough* Det 50 *cough* driver to reviewers to "benchmark with."

So I predict much of the same old same old from certain IHVs....sending of "use these special unreleased drivers to benchmark with" to review sites before being certified. And then deal with certification after the fact, if at all. (Ask forgiveness instead of permission.)

But certification is definitiely the "right thing to do", and if you do get buy-in from the IHVs, (and they adhere to it) that would be great.

It is also important that reviewers let us know if they have got new hardware and new drivers, but no confirmation about the usage of 3DMark. I think this is a very good thing, and we hope that the media takes the initiative to contact us if anything is unclear.

I think there's a much greater chance of this happening than IHVs proactively getting FMQL first. With all of the awareness surrounding driver cheats recently, I would hope that most reviewers would be keen on noting if the drivers are certified or not.

 

[Bjorn]

So I predict much of the same old same old from certain IHVs....sending of "use these special unreleased drivers to benchmark with" to review sites before being certified. And then deal with certification after the fact, if at all. (Ask forgiveness instead of permission.)

I don't really see that happen (or maybe i hope that this doesn't happen ). Nvidias drivers are now under the microscope and everything they release from now on will be investigated rather hard (which it should also). And releasing "reviewer drivers" with performance that doesn't match the afterwards officially released drivers would hardly be passed on without notification.

 

[Nick[FM]]

If you can pull that off....that would indeed be something. That is above and beyond my expectations actually. Glad to see you pursuing it! And yes, it will mean a lot of work for you, and it's appreciated.

It will be. Not "would" be. As said, it is in our target to make this happen.

I would have to disagree with that...or that (some) IHVs are paying little more than lip service to you on this.

Well, as these guidelines are still very new and fresh, we need to let the manufacturers "chew on them" for a while. I think all manufacturers agree that it is in the best interest of us all if we could "approve" (I hate that word) any unofficial drivers in cases like previews. Any such drivers would be approved for that one review only. Let's hope everything goes as it should.

I think there's a much greater chance of this happening than IHVs proactively getting FMQL first. With all of the awareness surrounding driver cheats recently, I would hope that most reviewers would be keen on noting if the drivers are certified or not.

Well, we hope that the media would be in contact even if they would get "FMQL" drivers with the package. Just to make sure that everything is ok. At least now in the beginning. Later on when everyone knows what to do, and what to believe in, I'm sure that reviewers can do it all without having to consult with us first. But it is always good to make sure, and we are here to provide help if needed.

 

[Joe DeFuria]

I don't really see that happen (or maybe i hope that this doesn't happen ). Nvidias drivers are now under the microscope and everything they release from now on will be investigated rather hard (which it should also).

I'd argue that nVidia's drivers have been "under the microscope" well before the Det 50 fiasco...and that didn't stop nVidia from releasing the Det 50s to the press. Has nVidia finally learned now? We can only hope!

And releasing "reviewer drivers" with performance that doesn't match the afterwards officially released drivers would hardly be passed on without notification.

In the enthusiast crowd...I agree. However, to the mainstream press mags that do little more than run a benchmark or two and proclaim the winner to the masses? Not sure...

 

[bloodbob]

Q: What do you mean by detecting 3DMark directly or indirectly?
A: The drivers may not include routines that in any way identify that 3DMark (or any part of it) is
running, and go into some special mode, instead of running 3DMark like any other 3D application.


Well one could argue that per shader replacement doesn't violate this as it doesn't go into a special mode nvidia this all the time.

 

[Joe DeFuria]

Q: What do you mean by detecting 3DMark directly or indirectly?
A: The drivers may not include routines that in any way identify that 3DMark (or any part of it) is
running, and go into some special mode, instead of running 3DMark like any other 3D application.


Well one could argue that per shader replacement doesn't violate this as it doesn't go into a special mode nvidia this all the time.

Depending on the shader replacement, that could violate other rules though. (Lowering quality, not being mathematically equivalent...)

Shader replacement is not categorically ruled out by these rules as far as I can tell. (Nor should it be.) Whether or not specific shader replacement is legitimate or not depends on the specifics of the replacement.

 

[ClyssaN]

I agree in general expect with this one:

Q: When will the enforcing begin?
A: The rules are in effect now. Our objective is to begin full-fledged enforcing at latest on October
31st. The reason is that we want to give all manufacturers time to adjust to the guidelines.

Months have passed, no need to wait this long ..., more and more users are being deceived, pull the drivers that are "cheating" and wait for the manufacturers to released new ones ...

 

[digitalwanderer]

Good answers!

If Futuremark can stick to it's new rules, I'll be willing to give it another chance. You guys have said all the right things, now it's just a matter of seeing how well the rules are enforced.

Good job and good luck.

 

[Bjorn]

Depending on the shader replacement, that could violate other rules though. (Lowering quality, not being mathematically equivalent...)

I would actually object a bit afa as the mathematically equivalent goes. Especially since we don't have any idea on what the card actually does internally. And if it doesn't lower IQ, how would you prove that it isn't/is mathematically equivalent ?

 

[Doomtrooper]

How can you detect if the driver is 'detecting' 3Dmark 03.exe when the driver is encrypted and doesn't allow a 'person' to see what is going on inside now.

Honestly, this is something that can't be checked anymore, and renaming the .exe won't help either as they may be detecting the shader in a more advanced way.

 

[Nick[FM]]

How can you detect if the driver is 'detecting' 3Dmark 03.exe when the driver is encrypted and doesn't allow a 'person' to see what is going on inside now.

Even if the driver is encrypted (or whatever) we have our ways to "fool" the driver. How, I can't say, but we have our internal tools.