Futuremark is full of shit.How can they bear this?

[Bouncing Zabaglione Bros.]

]Bouncing Zabaglione Bros. & co.,

Did you guys even read Tero's post over at Rage3D? If not, please read it carefully. As Tero wrote in his post over there:

This work has been time consuming and the issues are very complex and we have not wanted to comment on work in progress. We would like to apologize the time it has taken, but it has been time well spent and you will see the results soon.

Also:

We will publish clarifications and ground rules for our product usage on September 19th.

Yes I read that. And in the meantime you allow Nvidia to cheat and make public statements about your company and about your benchmark and you do nothing to refute the inaccuracies until months down the line. While you are not wanting to "comment" in the face of incorrect public perception of your product, your credibilty slips away hand in hand with Nvidia's disappearing credibility.

You were happy to jump down Nvidia's throat straight away with your first document against the Nvidia cheats, and now you keep quiet after Nvidia re-join your programme and issue "Futuremark approved" drivers to the public. Can you not see how that makes you and your company look to the public RIGHT NOW, not on September 18th when it's too late?

 

[digitalwanderer]

Worm and Patric, I'd like to thank you both for taking the time to post up and try and explain what is going on. I do understand that these things take some time to investigate and you do not want to make any allegations until you know what the what is...but by letting nVidia say that these drivers are FutureMark approved when they are in fact not is just WRONG on so many levels that I don't know where to start.

I appreciate you telling us we'll know more on the 19th, but I really think you're shooting yourselves in the foot by putting it off that long. What are you going to do on the 19th, announce the 45.23 set isn't acceptable?

Hmmm...Det 5's are supposed to be out 4 days before that....

If you can't respond to this in a timely matter it kind of kills the whole point of responding to it. You should NOT allow scores to be published until YOU approve the drivers, regardless of nVidia's agenda.

I'm very disapointed in FM's behavoir lately, and I'm not the only one. The net is a bit a buzz with disgruntled FMers, and every day just brings 'em more news that ain't good.

Without meaning to sound rude or cause offense, I gotta ask you two something: WTF?

 

[Nick[FM]]

Bouncing Zabaglione Bros.,

Making important decisions (in this case, clarifications and ground rules) isn't something we want to rush out with. We have been discussing with our Beta members about these things and getting a working and good solution is very time consuming. In a perfect world we would be able to make everything in 1 day, but sadly this just isn't possible. I am not really sure what it is that you are really looking for here?

digitalwanderer,

In what particular behaviour are you disappointed in? That we try our best to make this whole optimizations & benchmark thing work, and possibly set future ground rules for years to come? I understand that you guys are enthusiastic about things like these, and want answers in seconds, but trust me when I say that these kind of "operations" take a load of time to complete. We all would like to get this sorted out once and for all, and concentrate on making excellent future products in coperation with all our Beta members!

 

[Joe DeFuria]

]In what particular behaviour are you disappointed in?

Well, for me, I'm disappointed in learning that nvidia's assertion of having "Futuremark Approved Drivers" is patently false....and I hear about this a month after the drivers are released and the statement is made, and I hear about it in an unofficial manner.

I'm not personally asking for anyone to "rush to give approval or disapproval" to drivers or "optimization guidlines."

I do expect that when a company makes a claim, or even if a company doesn't make a claim but said claim is accepted as common knowledge, that the truth is told immediately if it is contrary to the claim.

A statement to the effect of "No, these drivers are not Futuremark approved at this time...we have not investigated them enough to reach a judgement. They are WHQL certified, so as per our current guidelines they will be accepted for the time being as valid until / unless we find reason to judge otherwise" would be sufficient for me.

All you have done by keeping quiet, is shoot yourself in the foot! (Which IMO you've been doing a lot of lately).

I simply don't understand why you would allow a myth to stand un-rectified? Especially one that makes FM look bad...drivers suppossedly being certified by you when it's suspected that they contain "illegal" optimizations?

 

[digitalwanderer]

]In what particular behaviour are you disappointed in?

Well, for me, I'm disappointed in learning that nvidia's assertion of having "Futuremark Approved Drivers" is patently false....and I hear about this a month after the drivers are released and the statement is made, and I hear about it in an unofficial manner.

I'm not personally asking for anyone to "rush to give approval or disapproval" to drivers or "optimization guidlines."

I do expect that when a company makes a claim, or even if a company doesn't make a claim but said claim is accepted as common knowledge, that the truth is told immediately if it is contrary to the claim.

A statement to the effect of "No, these drivers are not Futuremark approved at this time...we have not investigated them enough to reach a judgement. They are WHQL certified, so as per our current guidelines they will be accepted for the time being as valid until / unless we find reason to judge otherwise" would be sufficient for me.

All you have done by keeping quiet, is shoot yourself in the foot! (Which IMO you've been doing a lot of lately).

I simply don't understand why you would allow a myth to stand un-rectified? Especially one that makes FM look bad...drivers suppossedly being certified by you when it's suspected that they contain "illegal" optimizations?

Exactly what Joe said. (Albeit in a bit more "stompy/ranty" tone, I got me reputation to think of here too. )

EDITED BITS: How long does it take to make a statement on your site that nVidia is falsely saying that this set is FM approved? Is there ANY doubts or questions about that on your end?

 

[Bouncing Zabaglione Bros.]

]Bouncing Zabaglione Bros.,

Making important decisions (in this case, clarifications and ground rules) isn't something we want to rush out with. We have been discussing with our Beta members about these things and getting a working and good solution is very time consuming. In a perfect world we would be able to make everything in 1 day, but sadly this just isn't possible. I am not really sure what it is that you are really looking for here?

I'm looking for you to come out in a timely manner and support your benchmark and correct the falsehoods that are being perpetrated by Nvidia in the name of "Futuremark approved" drivers. I want you to say what the truth is ie, that current Nvidia drivers still cheat and so are not allowed in the ORB, that Nvidia's current drivers are not "Futuremark approved" as stated publicly by Nvidia, and that Nvidia has been contacted about this but has not issued a retraction.

You need to do this publicly in response to Nvidia's public falsehoods about you, and you can do that NOW without affecting working towards whatever your new guidelines are.

As it is, it just appears that Nvidia pushes Futuremark around and does what it wants - either because of it's size, threat of lawyers, or money it has given to you - and I'm not the only one that shares that perception. And I'm one of the minority of people that know and understand all about Nvidia's cheating drivers and what 3DMark2003 is designed to test. If I have no faith in your company, what do you think the "average Joe" is going to think after all the BS Nvidia has put out about your company and benchmark?

 

[Joe DeFuria]

As it is, it just appears that Nvidia pushes Futuremark around and does what it wants - either because of it's size, threat of lawyers, or money it has given to you - and I'm not the only one that shares that perception.

Agreed.

I honestly cannot think of ANY valid reason why Futuremark would not rectify the "Approved by Futuremark" assertion....other than "We don't want to p*ss off NVIDIA." And of course, we're left with our own imaginations to fill in the blanks with why that's the case.

An honest question: Can you offer an alternative explanation for not responding with the truth? I'm all ears.

I fully appreciate that Futuremark is between a bit of a rock and a hard place here. You are either going to p*ss off nvidia, or p*ss off the end users depending on the choices that you make.

I also understand that from a business perspective, you may feel that not p*ssing off nvidia is the wiser choice.

Once the decision is made however, don't blame the end-users for doing their damnedst to try and convince you that p*ssing them off is in fact, the wrong decision. Just as nVidia did their damndest to convince you (and succeeded) that p*ssing them off is the wrong way to go. So the next time you need to make a similar decision, you might have to re-evaluate the consequences of it.

 

[Quitch]

Just to let you know, I lost all faith in FutureMark and its products today. Guess you waited a bit too long to publish.

 

[digitalwanderer]

I fully appreciate that Futuremark is between a bit of a rock and a hard place here. You are either going to p*ss off nvidia, or p*ss off the end users depending on the choices that you make.

I also understand that from a business perspective, you may feel that not p*ssing off nvidia is the wiser choice.

Once the decision is made however, don't blame the end-users for doing their damnedst to try and convince you that p*ssing them off is in fact, the wrong decision. Just as nVidia did their damndest to convince you (and succeeded) that p*ssing them off is the wrong way to go. So the next time you need to make a similar decision, you might have to re-evaluate the consequences of it.

I'd just like to add that I ain't really sure it IS the best business solution for FM. Sure it might help them to keep nVidia happy and keep them checks rolling in, but it is tanking their credibility and without that no one will use their benchmark and their business will die anyways.

 

[Joe DeFuria]

I'd just like to add that I ain't really sure it IS the best business solution for FM. Sure it might help them to keep nVidia happy and keep them checks rolling in, but it is tanking their credibility and without that no one will use their benchmark and their business will die anyways.

Well, that's my point. Futuremark has to make a decision: potential negative publicity and loss of revenues from end-users, vs. potential negative publicity and loss of membership fees from nVidia.

Which is worse?

FM sould expect that end users will at least be trying to convince them that we are the "wrong party" to be p*ssing off from a business sense.

Whether or not that does impact their business more than p*ssing off nvidia remains to be seen. I'd like to think that keeping nvidia (or any IHV) happy is a short term gain for long-term pain, because I think keeping IHVs happy is not the ideologic purpose of a benchmark. The purpose is to be fair, such that end-users can rely on it. Keeping IHVs happy should be secondary to ensuring the integrity of the benchmark.

And if it's not possible for a business to succeed with that model...then I just assume the business not exist, because it can only do more harm than good.

 

[digitalwanderer]

I'd just like to add that I ain't really sure it IS the best business solution for FM. Sure it might help them to keep nVidia happy and keep them checks rolling in, but it is tanking their credibility and without that no one will use their benchmark and their business will die anyways.

Well, that's my point. Futuremark has to make a decision: potential negative publicity and loss of revenues from end-users, vs. potential negative publicity and loss of membership fees from nVidia.

Which is worse?

FM sould expect that end users will at least be trying to convince them that we are the "wrong party" to be p*ssing off from a business sense.

Whether or not that does impact their business more than p*ssing off nvidia remains to be seen. I'd like to think that keeping nvidia (or any IHV) happy is a short term gain for long-term pain, because I think keeping IHVs happy is not the ideologic purpose of a benchmark. The purpose is to be fair, such that end-users can rely on it. Keeping IHVs happy should be secondary to ensuring the integrity of the benchmark.

And if it's not possible for a business to succeed with that model...then I just assume the business not exist, because it can only do more harm than good.

My bad Joe, you're right and me post was a bit redundant.

My apologies, I'm just a bit chagrined and outraged at FM's apathy and their lame "Well we really need a bit of time to deal with this" attitude.

 

[Bouncing Zabaglione Bros.]

Whether or not that does impact their business more than p*ssing off nvidia remains to be seen. I'd like to think that keeping nvidia (or any IHV) happy is a short term gain for long-term pain, because I think keeping IHVs happy is not the ideologic purpose of a benchmark. The purpose is to be fair, such that end-users can rely on it. Keeping IHVs happy should be secondary to ensuring the integrity of the benchmark.

And if it's not possible for a business to succeed with that model...then I just assume the business not exist, because it can only do more harm than good.

The perception of a benchmark being fair is very important to users, although I suspect the payment from is IHV's is actually what pays for the benchmark to be developed. However, you have to ask yourself, what is going to happen when the user perception is that the benchmark and Futuremark can no longer be trusted?

Sure, Futuremark will still be getting money from Nvidia, but why will any other IHV stay with Futuremark if Futuremark continues to show a lack willingness to support their benchmark in the face of a cheating member? What happens when the public have no faith in Futuremark, and companies like ATI ( and eventually Nvidia) see no reason to pay for a benchmark that the customers (including OEMs eventually) have no faith in and stop using?

 

[digitalwanderer]

Whether or not that does impact their business more than p*ssing off nvidia remains to be seen. I'd like to think that keeping nvidia (or any IHV) happy is a short term gain for long-term pain, because I think keeping IHVs happy is not the ideologic purpose of a benchmark. The purpose is to be fair, such that end-users can rely on it. Keeping IHVs happy should be secondary to ensuring the integrity of the benchmark.

And if it's not possible for a business to succeed with that model...then I just assume the business not exist, because it can only do more harm than good.

The perception of a benchmark being fair is very important to users, although I suspect the payment from is IHV's is actually what pays for the benchmark to be developed. However, you have to ask yourself, what is going to happen when the user perception is that the benchmark and Futuremark can no longer be trusted?

Sure, Futuremark will still be getting money from Nvidia, but why will any other IHV stay with Futuremark if Futuremark continues to show a lack willingness to support their benchmark in the face of a cheating member? What happens when the public have no faith in Futuremark, and companies like ATI ( and eventually Nvidia) see no reason to pay for a benchmark that the customers (including OEMs eventually) have no faith in and stop using?

I wonder what ATi's take on all this is, as well as nVidia's? Has anyone dropped them a friendly line yet to check?

(The Dig scampers off happily to his e-mail client for a friendly letter to ATi and another quick go-rounds with nVidia.... )

 

[Joe DeFuria]

The perception of a benchmark being fair is very important to users, although I suspect the payment from is IHV's is actually what pays for the benchmark to be developed.

Agreed on both counts.

And this is where Tommy's (and others) points about FM's business model being a conflict of interest come into play. If your benchmark business model is such that you rely on IHV money to be sustainable, I think that's an inherently bad model.

I see nothing wrong with charginging IHV fees for participation. After all, there are costs involved with giving IHVs controlled access, feedback programs etc. However, I see lots wrong with a business model that relys on those fees to be viable.

What happens when the public have no faith in Futuremark, and companies like ATI ( and eventually Nvidia) see no reason to pay for a benchmark that the customers (including OEMs eventually) have no faith in and stop using?

What FM is counting on, IMO, is that any single IHV is afraid of the risks of being the "first one out." Case study: nVidia.

Nvidia drops from the program. Now, nVidia is thinking "what if despite our public objection, 3DMark remains the industry standard benchmark...and now we've lost access to it?" That's a very high risk issue, so it's no wonder nvidia got back in as fast as they could.

Now nVidia is back in, and ATI might be feeling slighted by it. Should ATI drop out on principal? Well, I'm sure ATI is thinking..."what if we drop out, lose access, and 3DMark continues to be an industry standard?"

So, the IHVs are more or less hostage, as long as there is a reasonable chance of 3DMark remaining an industry standard benchmark.

What needs to happen, is that ALL IHVS more or less simultaneously say "enough is enough." They all get together and say "we're sick and tired of this optimization game, spending resources, etc., that we don't feel we're getting good enough return on...so we're ALL dropping out."

That will either force 3DMark to go away, or will force 3DMark to change its business model. The latter is more preferable to me.

As you aluded to, one way for IHVs to see the value in their efforts decrease, is for the "trustworthiness" in the bechmark to visibly erode. However, this is not as simple as a bunch of enthusiasts shouting this from the top of their lungs. It would take OEMs saying "I don't care what the 3DMark score is anymore....I don't believe it helps me sell the product."

 

[digitalwanderer]

However, this is not as simple as a bunch of enthusiasts shouting this from the top of their lungs. It would take OEMs saying "I don't care what the 3DMark score is anymore....I don't believe it helps me sell the product."

But don't the OEMs rely on FM because the enthusiasts approve/endorse it? Won't they tend to follow what the enthusiasts do, albeit a bit slower?

 

[Joe DeFuria]

However, this is not as simple as a bunch of enthusiasts shouting this from the top of their lungs. It would take OEMs saying "I don't care what the 3DMark score is anymore....I don't believe it helps me sell the product."

But don't the OEMs rely on FM because the enthusiasts approve/endorse it? Won't they tend to follow what the enthusiasts do, albeit a bit slower?

Yes and no. Ideally, yes, that can happen.

However, if the "3DMark Brand" is strong enough, enthusiasts might believe it to be crap, but the masses might not. And the OEMs will cater to what the masses want....right or wrong.

The "easiest" way for one brand to lose market favoritism, is to have a competing brand replace it as the market leader. For example, if Aquamark3 gains momentum and accpetance (for whatever reason), then this can displace 3DMark, and become the OEMs next "darling."

If there is nothing to compete with 3DMark, it becomes harder and harder to just drop it.

 

[digitalwanderer]

The "easiest" way for one brand to lose market favoritism, is to have a competing brand replace it as the market leader. For example, if Aquamark3 gains momentum and accpetance (for whatever reason), then this can displace 3DMark, and become the OEMs next "darling."

If there is nothing to compete with 3DMark, it becomes harder and harder to just drop it.

Let's just hope that Aquamark3 and the HL2 benchmark take their crediblity a lot more serious than FutureMark apparently takes theirs then.

 

[jb]

Again I would like to stress that absolutly no optimizations should be allowed in a synthetic benchmarks. I dont care about how games work as If I want to now that I will used games. I agree with the Rage3d article; a becnhmark gives out a set task with a set work load. Any wiggle room there = bad.

 

[Doomtrooper]

Exactly why Doom 3 should not be used as a benchmark...the old 'workload' example applies there too. Yet when the game releases the entire internet community and review sites will be filled with Doom 3 benchmarks, the best 'DX9' benchmark

 

[Joe DeFuria]

Exactly why Doom 3 should not be used as a benchmark...the old 'workload' example applies there too.

Yes and no.

The old "which is the best card to play Doom3" also applies to Doom3. So Doom3 can and should be used as a Doom3 benchmark. No one "plays" 3DMark, so I of course agree that no workload differences should exist in a benchmark fundamentally designed to stress GPUs.

Yet when the game releases the entire internet community and review sites will be filled with Doom 3 benchmarks, the best 'DX9' benchmark

Unfortunately, you are probably right. This is the fault of reviewers having their collective heads up their asses though. Not because it's not valid to use Doom3 as a benchmark in any capacity.