Ops,"I" did it again. Why does Futuremark still ke

[micron]

I'm saying that it sux to have a new videocard and not be allowed to submit scores, after you've paid for the pro version.
It doesnt really matter now because FM fixed the problem
The scores are being submitted, and everyones happy

 

[digitalwanderer]

FM needs to put a clause in their 'Pro' purchase contract that warns of score submision refusals if any of the IHV's become cheating bastards.

Yeah, but I was talking about the bit in their policy where they state they only accept results from officially released WHQL drivers.

 

[micron]

FM needs to put a clause in their 'Pro' purchase contract that warns of score submision refusals if any of the IHV's become cheating bastards.

Yeah, but I was talking about the bit in their policy where they state they only accept results from officially released WHQL drivers.

I know you were, and I understand FM's policy.
The pro version end-users were still complaining constantly, and it was something that FM had to deal with. And they did.

 

[digitalwanderer]

FM needs to put a clause in their 'Pro' purchase contract that warns of score submision refusals if any of the IHV's become cheating bastards.

Yeah, but I was talking about the bit in their policy where they state they only accept results from officially released WHQL drivers.

I know you were, and I understand FM's policy.
The pro version end-users were still complaining constantly, and it was something that FM had to deal with. And they did.

I don't follow FM's boards, how did they deal with it?

 

[micron]

They got back in bed with Nvidia.

 

[digitalwanderer]

They got back in bed with Nvidia.

Ok, I knew that. (I guess I HAVE been following it a bit better than I thought. )

 

[WaltC]

ouch.

Actually, in thinking a bit more about the conflict of interest inherent in the business model, it strkes me there are actually two major conflicts of interest there.

The first is that fact that while the assumption of the IHV partners is that they are paying for software which can be used as a marketing tool to portray their products in a positive light, the public is thinking they are using software which provides a level playing field for an ojective evaluation of the features supported by differing 3d hardware in a comparative context.

The second is that you have competing IHVs paying dues, but expecting entirely different things from the software. For instance, nVidia pays x dollars and tells FM to "go this way" with its software, and ATi pays FM x dollars and tells them to "go that way" with its software. So what does FM do? If they go "this way" they risk alienating ATi and having ATi quit the program, but if they go "that way" with it they risk alienating nVidia and having nVidia quit the program (which has already happened.) Last, if they take the middle route and split the difference between them then they risk alienating both of the IHVs and possibly having both of them quit the program.

Pre-R300 I'm guessing that conflict of interest between IHVs was not that much of a problem since nVidia had a pretty good lead at the high end. But now that ATi is emerging as dominant, I can see several bones of contention developing along the lines that prompted nVidia to quit last year. As the competition intensifies between the IHVs my thought is that the situation isn't likely to subside, but rather to grow more acrid.

The funny thing about companies is when they shell out money for programs like this they naturally expect some tangible benefit in the way of goods and services in return. If they don't perceive they are getting such benefits they lose the motivation to pay for the privilege of participating. Looks like a thorny problem for FM going forward.

I think the best solution therefore would be for them to change their business model and deal with the IHVs on a voluntary basis through NDAs and drop them from the partnership dues-paying rolls. They could maintain their partnership with oems like Dell (and other entities) for the purposes of sales and distribution, and concentrate on creating benchmark software for sale to the general public, primarily to serve the general public. If an IHV wanted to see its hardware represented in a forthcoming FM benchmark it could supply whatever help it wanted to FM, sans payment, and FM could derive its income from its non-IHV partners and sales of its software to the general public. That way FM could eliminate these potentially serious conflicts of interest entirely. If they don't do something like this I can see these conflicts eventually tearing the company apart. Just my opinion, of course.

 

[Mark]

only Walt could take a one word post and post and reply with an editorial (ok, maybe demalion too)

 

[StealthHawk]

We are all also studying if such "optimizations" should be app-specific or application-wide (generally, but even "generally" can be an important word).

rev nvidia themselves stated on those slides that these "optimisations" (which will forever be in quotations) must accelerate more than just a specific application. I really don't understand how FM can allow such an app specific hack when nvidia promise otherwise.....

The slides say no such thing. One of the guidelines is that an optimization must accelerate more than a benchmark. I can only assume that this applies more to games with a benchmark mode, rather than saying that no optimizations are allowed in synthetic benchmark programs period.

45.23 was supposed to be the first driver to implement these guidelines...I for one have not heard of any changes.

 

[Malo]

The slides say no such thing. One of the guidelines is that an optimization must accelerate more than a benchmark. I can only assume that this applies more to games with a benchmark mode, rather than saying that no optimizations are allowed in synthetic benchmark programs period.

I see what you mean yes. But it's still a little ambiguous.

If Nv replaced a shader in 3DMark03 and used the same shader replacement in a game, does that constitute as a valid optimisation?

 

[WaltC]

One of the guidelines is that an optimization must accelerate more than a benchmark.

The way I interpret that is to mean that they'll no longer be making performance-increase claims for driver sets which do nothing except to "accelerate" benchmarks. Or, looking at it another way, they are saying they won't be optimizing strictly for benchmarks anymore, such as 3dMk, for instance.

Obviously this latest Detonator set has flunked the new guidelines test, so I await with relish the moment when nVidia actually does what it says it is going to do...

 

[digitalwanderer]

Obviously this latest Detonator set has flunked the new guidelines test, so I await with relish the moment when nVidia actually does what it says it is going to do...

I'm waiting with a healthy dose of skepticism & suspicion.... :?

 

[Quitch]

Further discussions have shown, that under some circumstances (that we're trying to define at the moment) shader replacements can be accepted.

I hear the funeral march playing in the distance...

 

[tEd]

Further discussions have shown, that under some circumstances (that we're trying to define at the moment) shader replacements can be accepted.

who said that?

 

[FUDie]

Further discussions have shown, that under some circumstances (that we're trying to define at the moment) shader replacements can be accepted.

who said that?

Patric Ojala in this post.

-FUDie

 

[WaltC]

Further discussions have shown, that under some circumstances (that we're trying to define at the moment) shader replacements can be accepted.

who said that?

Patric Ojala in this post.

-FUDie

Heh... Further discussion:

nVidia: We want to use our own shader code.

FM: Hmmmm....OK, uh, how to say this delicately...? Ummm....we haven't received your sign-up fee as of today....it's 3 p.m., our time....

(Two hours later)

nVidia: OK, did you get the PayPal receipt?

FM: Yes. In the bank.

nVidia: So?

FM: After lengthy discussions we've determined that yes, under some circumstances, it will be OK for you to use your own shader code--yes, certainly.

nVidia: For Pete's sake, what circumstances?

FM: Well....ummm....before we divulge that, there's the little matter of a slight...shall we say...shortfall?...in your initial deposit of trust with us...

nVidia: I don't believe this...

FM: Yes, well...Believe it, believe me...

nVidia: *sigh* Same PayPal account...?

FM: Well, you see we have a delicate matter of taxation in our country to consider and the additional....uh, contribution...cannot go through PayPal. I want you to call Pytor--his first name will be sufficient--at 555-1212, and you are to say, "The rose is red today," and he will answer, "My mother lives in Cambodia," in reply. If the proper phrase is not returned immediately *hang up the phone and get out of there.* Otherwise, Pytor will call later with instructions on what you must do. I would advise waiting by your telephone until Pytor rings you up...Once "the package" has been received, then you will receive clear instructions from me on what to do with your shader code....And, trust me, if Pytor is thwarted in any way I will only tell you where you can stuff your shader code, instead. I hope we understand each other...

nVidia: We get the picture, all right!

FM: I'm gratified. Nice doing uh, the business with you!

 

[micron]

Thats quite a picture you've painted there Walt....

 

[Quitch]

ROFLOL!

 

[just me]

Patric,

I just read your post in your personal forum here at B3d, and I've read your comments in this thread as well. The following was among the most troubling of your remarks, and the most disappointing to me personally:

...Immediately as Nvidia left our development program, getting them back into the program was raised to a very high priority. Our highest priority naturally is and remains the development of high quality and impartial benchmarks.

I'm trying to get you readers to understand what a great thing it is for us that Nvidia re-joined our development program. I think this is the best thing our company has achieved since we launched 3DMark03. The Futuremark employees went through some tough times back there, but we're all happy to get Nvidia back, including AJ (even though some weird rumours tell differently) .Immediately as Nvidia left our development program, getting them back into the program was raised to a very high priority. Our highest priority naturally is and remains the development of high quality and impartial benchmarks.

I have to tell you I find this statement simply baffling. nVidia belonged to the 3dMk03 development program for at least 15 months out of the 18-month 3dMK03 development cycle (as they didn't quit the program until December of '02.) What was it, precisely, that you could obtain from nVidia by "getting them back" into the program that would allow you to make 3dMK03 a "high quality and impartial" benchmark that you failed to obtain from nVidia during the long months in which it belonged to the FM program during the 3dMK03 development cycle? I simply cannot imagine what that might be. Please elaborate.

nVidia was in the 3dMK03 development cycle long enough to learn how to cheat the benchmark with dispatch when it shipped--it boggles the mind how you might think that getting nVidia back had anything at all to do with making 3dMK03 a "high quality and impartial" benchmark, since presumably that's what it was when it shipped.

How is nVidia being in or out of the program relevant to 3dMk03 being a high-quality benchmark as you define it, since nVidia was a full partner for virtually the entire development cycle of 3dMK03?

Hopefully, you are not declaring that 3dMK03 when it shipped was anything but a high-quality and impartial benchmark...? This is baffling for me, honestly.

I'm trying to get you readers to understand what a great thing it is for us that Nvidia re-joined our development program. I think this is the best thing our company has achieved since we launched 3DMark03. The Futuremark employees went through some tough times back there, but we're all happy to get Nvidia back, including AJ (even though some weird rumours tell differently) .

It should be obvious why Nvidia is so important for our development program, but I'll repeat some of the key reasons for those unfamiliar with our company's business.

In order to produce top quality forward looking benchmarks, we need the input and cooperation of all the major players of the industry.

So, how is it you did not receive "input and cooperation" from nVidia for 3dMK03 development when nVidia was paying you for the privilege of providing such throughout the 3dMK03 development cycle? I'm sure I don't have to remind you that the events which occurred prior to nVidia leaving your program, prior to 3dMk03 shipping, prior to nVidia cheating the benchmark, and prior to your audit report detailing exactly how they cheated it--prior to these things nVidia was just as much a part of the FM program as it is today.

Your supposition seems to be that you require nVidia's participation to be able to write a good benchmark, and yet you had nVidia's participation when you wrote 3dMK03 and before *nVidia* made the elective decision to pull out and discredit your company's software.

So if nVidia was unable to assist you in writing a high-quality, impartial benchmark prior to resigning from the program, what has changed so that now nVidia's participation is *required* to produce that result?

How could we otherwise get the highly confidential information of where the industry is going next? Each 3DMark version is aimed at the new if not next generation of hardware, and it is of crushial importance to have comprehensive information of that next generation, in order to make a benchmark for it in advance. In addition to making just high quality benchmarks, they must be impartial, and it is not convincing to release an impartial benchmark developed in cooperation with just one of the major IHVs. If we would not have got Nvidia back, we would have still done our best to optimize it also with Nvidia's upcoming products in mind, but it would have been way harder.

Again, this does nothing to explain the puzzling issue you've raised, which is why you failed to obtain that "comprehensive information of that next generation, in order to make a benchmark for it in advance" from nVidia during the development of the 3dMK03 software (since nVidia belonged to the program for the bulk of the development cycle.) Of course, the truth is you did obtain it, but that nVidia wasn't happy with what you did with the "comprehensive information" and so it quit your program. That's my take on it based on what you've said.

Of course you've glossed over the very odd structure of your business model--which is that companies pay you for the privilege of providing their own next-gen information to you so that ostensibly that information will be used in the construction of a benchmark which will portray their products in a positive light and help them sell those products. Hopefully, you can see the glaring conflict of interest in doing that and in providing a "high-quality, impartial" benchmark for the 3d-card buying public. To that end I can see nVidia's point in quitting--their view was that they were paying you for specific services which you were no longer providing them. It's the conflict of interest inherent in your business model which has caused all of this to happen, IMO.

Sadly, I must regrettably conclude that nVidia's participation is a requirement for your company because of nVidia's financial contribution to your company based on the membership fees it pays you. Obviously, I guess you've agreed to start giving nVidia its money's worth and nVidia is now back in. Seriously, unless you guys start doing some serious "rethinking" about your business model and understand the inherent conflict of interest within it I don't see much of a future for you in this endeavor, and I see more of this same sort of trouble for you on the horizon.

Ditto, ditto, ditto Walt! 8) Nice to see I'm not alone in 'seeing it' the same way. FM can't have it both ways > either it shipped under false pretenses or it is a valid bench & doesn't need changing. Seems FM is admitting it shipped under false marketing pretenses & is not what they represented it as originally.

I'm also aghast at the claim of 'we didn't know'. My gawd they're in the business to know & if they don't > then they shouldn't release a product for sale w/the claims they do.

.02,

 

[Hellbinder]

You know the problem here? The problem is that Nvidia is going to get various developers to write specific code to support their short commings. Id like to know when having the cash and influence to cover up shortcommings suddenly Transformed into what we all want to accept as the Golden Standard?? Also being included in so called *industry standard benchmarks*??

This is just Ridiculous. Because the *kinds* of optimizations that ATi made, and would make are not the same endless ream of *CHEATING* BS that Nvidia is going to pull. FX12 is *NOT* part of the DX9 spec. Which is what they are going to do. Among other Questionable practices. All,, I repeat *ALL* in an effort to give them the best 3dmark03 score. So they can have the bragging rights and help line their cofferes. Does it matter that they get it Honestly?? Hell no. Becuase now everyone is Cow-Towing to them Handing them the *Undeserved* Crown on a Golden Platter. In a few months when the PCmags of the World crown Nvidia as the Performance Champion, and the "Card to own" based on 3dmark03 alone (becuase Nvidia endorses it now).. Not one comment will be uttered about *How* they suddenly came out so far on top.

Honestly, I hope that ATi optomizes the **** out of 3dmark03 now. I mean to Ridiculous levels. And overrides any user changes that can be made. Or, Lets Put in a Driver Setting called *Runny soup* that can be used for *all games* that way it passes as a legit *optimization*. I say, Futuremark wants to play the Nvidia Game Fine. Then lets Turn ATi's Rendering of 3dmark03 into a bowl of Runny Soup

(I am a little Irritated right now.. so please bear that in mind)