Futuremark Announcement Delayed

They already said all of this when they caught Nvidia cheating. Of course we now what happened then. They caved to the pressure from Nvidia and reversed themselves. What's to say they won't reverse themselves again if they catch another high dollar IHV cheating and the IHV applies a little pressure?

Basically, I don't see the point in releasing these guidelines. They already stated these same things before and completely ignored them.

 

So what about the 45.23 drivers you complained about with nVidia? What about the results obtained with that drivers? Are you (you = Futuremark) gonna to say something official about it, or you'll wait 'til nVidia make available a new release like 51.xx?

Bye!

 

Unless Futuremark release an updated version which is NV3x friendly, nVidia are pretty much banned from 3dMark now - assuming those rules are enforced, and aren't loop hole filled.

But of course, in reality, nVidia aren't going to get rid of all the cheats which break those rules, since their score will absolutely plummet. Time will tell on this one.

 

I pretty much thought the same thing. Publishing these new 'rules' is all fine 'n dandy; enforcing this rules is what will separate the wheat from the chaff.

 

Yep, I see that happening shortly. Futuremark will release a patch that either has an nVidia specific codepath (thus doing the cheating for nVidia), or simply does what Massive did with AquaMark3 by using _pp hints for every single shader in the benchmark. nVidia gets to keep their score without having to resort to driver cheating (not that it would bother them) and FM is happy because the nV cash spigot remains on.

 

Futuremark using _pp hints in places where full precision capabilities are not fully utilized (which does not quite resemble the places nVidia has been using it) wouldn't be "cheating for nVidia", or (as you seem to imply) anything "shady" or undesirable, even though the R3xx has no need to pay attention to them. Those hints are part of the DirectX spec, and DirectX is the focus of the benchmark.

However, neither would the introduction of _pp hints be a believable reason for NV3x scores remaining unchanged from the figures web sites are still publishing based on 3dmark 03.

EDITED: for clarity

 

Um. Guys? We've got 7 pages of this thread where everyone is going "damn, those guidelines better be good, or else...". Now that we have the guidelines (and they at least seem to be what I was hoping for them) everyone is going "these guidelines mean nothing, they better enforce them well, or else...".

As far as I can figure out they're pretty much exactly what I was hoping for? Would anyone care to comment on the actual guidelines themselves?

- Tom

 

It's exactly what I was hoping for too to be honest, but now that they've stated their rules the inevitable question of enforcement does arise and there still are a lot of questions with that.

It's a good statement, provided they stick to it.

 

Generic optimizations that do not violate the above rules and benefit applications in general are acceptable only if the rendering is mathematically consistent with that of Microsoft® DirectX® reference rasterizer.


Define consistent please.

I see nvidia with their lawyers driving a semi though legal holes.

 

I'm 100% with you on that. Did you notice the sentence: "We will cooperate with all major graphics vendors to enforce the guidelines as quickly as possible." That has to be dry humor, as I'm having issues with believing that Nvidia will 'co-operate' with getting app-specific optimizations away from their drivers.

BTW: Speaking of Nvidia, what do you expect them to do? Do you think they'll pull away from Futuremark's beta program now and begin to bash 3DMark again? I don't believe for a second that they'd actually clean-up their drivers so I'm a bit baffled here? Especially if Futuremark actually will begin to enforce those guidelines (altough that remains to be seen...) it will be interesting to see how Nvidia reacts. (pun intended: "It seems to us that Futuremark has deliberately tried to stop us cheating" doesn't sound great on a press release...)

- Tom

 

There seems to be too much wiggle room in it for me. And for it being what it was it took way to long.

It reads to me like nvidia and futuremark expect things to go back to the way they were and I dont think we can let that happen (I know I can't.)

With this kind of PR I dont expect things to change much at all unfortunately. There needs to be a bigger push to inform the public, a policy on bugs and a stated timeframe on both.

Id love to be one of those people who can just dismiss futuremark/3dmark but I cant. I waited and now from this point on I am personally going to demand much more from all parties involved including B3D.

 

Hmm... I have hard time believing that Nvidia would like these guidelines. They're very clearly against application detection, which by its own right should strip away most of the cheats. They also do not accept lowering filtering modes (trilinear downgrade a'la UT2k3), so it doesn't strike me as something that Nvidia would be happy with.

- Tom

 

I would like to see a form of punishment in place whereby if a IHV is found to be cheating its drivers containing the cheats would be disqualified. Furthermore the next driver release from that company should automatically have its score reduced by 10%-20% for period of time. Consider it a form of probation. This would give Futuremark and others time to investigate and would prevent IHV’s from continually releasing drivers for the purpose of circumventing the guidelines.

 

Pagh, you're way to kind....I'm much more in favor of public flogging of the offending companies CEO or something similarly effective.

 

Well, I don't think this is a good idea. The penalty idea just gives bad info to pre-empt other bad info. That just adds confusion and works to dilute the foundation of good results on Futuremark's part, and gives valid reasons for people to continually view their results with doubt when they should be working towards the exact opposite occurrence.

Joe's and Tommy's comments in the other thread cover the basic idea of enforcement pretty completely, except that I think some of it is not realistically sustainable against the types of tactics nVidia has shown a willingness to use (and achieved some success with, even while Futuremark had a hard line enforcement approach).

...

I'm all for pointing out which drivers have failed their guidelines, and I'd propose it might very well be a required first step to regain some measure of consumer trust in the face of the amount of bad information associated with 3dmark 03 that is out there.

What I think is important to make distinct (both when discussing and proposing such to Futuremark, and for Futuremark's dealings with IHVs moving forward) is the idea of what to do now to start to recover trust, which would be a temporary and directed action to correct the current situation, and what to do afterwards as a policy to defend and maintain that recovery.

I think first we need for Futuremark to recognize the situation from the individual consumer perspective, whereas I currently think they view many recommendations in terms of ongoing policy only, instead of distinctly considering a more drastic "course correction" separate from a "steady course" from then on out. Much of what has been proposed about the rules might benefit from being addressed more clearly as directed towards one of these or the other.

My thoughts directly relating to ongoing policy:

It doesn't seem feasible to maintain an ongoing policy of pointing to bad drivers and results, but what might be is the much less onerous (and less "political") one of auditing driver result ranges for a card for a current driver revision, and then certifying them (perhaps quarterly, though monthly would be better if possible). It is much easier to defend granting approval than to be actively pursuing condemnation...let the consumer confidence in what "certification" means to allow them to have the knowledge to decide when they should have reason to condemn IHVs.

The comments about Futuremark discretion in evaluation need to apply to this, and places Futuremark's interest in such discretion more perceptibly (to consumers) in line with defending the honesty and reputation of the benchmark. Futuremark needs to work to formalize and convey reason for confidence in that discretion...I think a certification methodology achieves that without adding the further burden of trying to go toe-to-toe and "penalize" billion-dollar companies head-on on an ongoing basis. Also, an option for audits conducted on request, along with extended testing (perhaps involving certain BETA members without conflicting interest, I think would make sense) for audits, limits the expenditure on thorough investigation (when signficant departures in performance results occur) and all the attendant effort that might entail, while still protecting the consumer even when the efforts aren't presently occurring.

I'd initially hoped for exactly this type of certification system, but self maintaining (by consumers), for image quality in the image quality/driver version database idea I'd proposed for 3dmark at the beginning of the year, but in light of the education I (and most of us, I believe) have received about the measures an IHV may use for deception in benchmarks, extending it to drivers and performance result ranges seems to be the simplest additional step.

Of course, offering better tools in their benchmark could still serve to make the self-regulation by users more feasible again. This is a factor I think it important for Futuremark to consider as well.

In any case, I think the movement towards "certified" results and "uncertified" results has already been demonstrated as necessary, that the ORB database could be modelled on the concept with limited modification, and could serve to facilitate this mechanism while not impeding consumer usage for timely hardware comparison (a foundation of Futuremark's "enthusiast" popularity which I don't think should be forgotten). Perhaps even an additional label, such as "overclocked" (or perhaps something a bit more "inspired"/"catchy", like "extreme" or whatever phrase conveys some positive connotation but refrains from implying that they are "verified") for results that exceed certification criteria concurrent with higher card frequencies being recorded at benchmark time, to address the enthusiast appeal that serves/served to maintain the benchmark's popularity.

...

Thoughts?

 

Gah! I wish I could submit an edit without it timing out and doing nothing, so I could clarify some of the wording.

 

Gah! I wish I could submit an edit without it timing out and doing nothing, so I could clarify some of the wording.

 

Not even pp can save NVIDIA.

re: AM3, take a look a scores with the 44.03 driver. Isn't it odd that 45.23/51.75 almost doubles the performance on 44.03? Something is rotten in the corporate building of NVIDIA.

 

Demalion,
I think you posted the wrong link in your thread. I could not see any comments by Joe or Tommy (Azbat ?).
http://www.beyond3d.com/forum/viewtopic.php?t=8061

The point I was trying to get across was that there should be a mechanism in place that discourages IHV from releasing drivers every time they are caught cheating just to stay on top of the ORB. While these new drivers may contain cheats and latter be identified, considering the lag time, they will still have allowed the IHV to gain mind share. I look at it like a retraction in a newspaper. I rarely undoes the damage.