Ops,"I" did it again. Why does Futuremark still ke

[CorwinB]

Wörm told me there is an interesting discussion going on here, since I'm a bit lazy to follow all forums personally

As usual, I'll comment in the nice personal forum Dave and the guys gave me (thanks again). Check it out in a few minutes, I'll start typing right away.

I've just read your interesting post in your personal forums, and I have a few questions left :
a) what are (if you can answer) your "gut feelings" regarding the "optimizations" in drivers
b) where will 3DMark 2003 go from here ? I agree that NV rejoining the Beta program can be a good thing, but only if they play nice. If they continue with the various "optimizations" while your other partners are denied the same thing, then you have to agree that the benchmark loses all meaning.
c) What does FM intends to do to convince people that their benchmark still is relevant ? Will you change your stance of what "optimizations" are acceptable ? How will you check that IHVs don't pull dirty driver tricks ? You mention a large number of times how happy you are that NV is rejoining. But you seem to forget (or you dodge the issue) that there is another thing you need besides major IHVs participating in your beta program : you need the public perception that your benchmark is an useful tool for evaluating video adapter performance. And, as of now, this is not the case : some people (mostly due to Nvidia bashing 3DMark and synthetic benchmarks, which they still have to apologize for) now have a very [h]arsh stance against 3DMark, and other, while still being convinced that synthetic tools are useful, also don't believe in 3DMark anymore because of the 180° on the "C" word and the lack of communication regarding driver updates since... For many people, FutureMark seems to have absolutely zero pride regarding its products and their butchering through various "optimizations". How do you plan to adress this ? What about FM developing a demo for NV as was rumored ?

Edit : just seen your post

Also, the whole shader replacement thing was a brand new issue.

While *shaders* on personal computers are technically a new thing, "cheatoptimizing" in benchmarks (any benchmark, not just GFX) is not. You would think a company whose entire business model is supposed to rely on accurate benchmarking tools would have been prepared...

 

[digitalwanderer]

we need to be very careful about what public statements we make. This industry is full of eager writers, who seem to spend their days trying to read between the lines and dig out some issues from anything and everything that is published.

And here I didn't think anyone noticed me, I'm touched!

 

[madshi]

...

Thank you for your comment. I understand the problem and I understand your situation. I just hope that you get it all sorted out soon. Because right now the order of the graphics cards in your hall of fame is somehow "wrong" (if you know what I mean ). And that's probably the reason why lots of people currently consider 3DMark03 as being invalid - which I find a pity...

 

[swanlee]

I think you guys are opening a huge can of worms here by agreeing to Nvidia's terms about optimizations and that your original stance of no cheating allowed ever was the proper response.

You guys put out a GPU Benchmark based on DX8/9 API and that is what you should stick to. By allowing any optimizations you open your program up to the fact that games might not be optimzed the same way or considering the past that these might open up the fact that a crappy card can do well with specific optimizations in the dirvers.

Your program is a benchmark and any optimizations are for the sole purpose of increasing the benchmark score which may or may not be reflective in any games.

You guys need to stick to the DX API' standards and gear your benchmark toward these standards. And if another companies hardware performs badly using DX standards then that is just plain tough most games will be geared using these standards and not developed specifically for one vendors cards.

You are ruining the integrity of your product by allowing this.

 

[Tridam]

First of all, we need to be very careful about what public statements we make. This industry is full of eager writers, who seem to spend their days trying to read between the lines and dig out some issues from anything and everything that is published.

This industry is also full of 'false marketing truths'. It's not always easy to deal with them. We need to read between the lines to separate true facts from marketing facts. We need to do this with every official statement.

Of course, quoting a board post from someone who works for company xxx and trying to dig out issues from it is really unfair. But what must we do when some board posts are very different from official statements? We can't ignore them…

If we don't immediately comment some new turn in the industry, it might be due to the fact that:
a) We don't know yet the whole extent of the issue and we're still trying to get to the bottom of it. This is usually what's happening when we don't react to something the very next second.
b) We see that the issue in question as not ours to comment. Our goal is to provide the industry with tools for hardware and driver testing. Analyzing the results and judging what's good or bad is what the professional hardware press is for.
c) The whole issue was raised due to some wild rumour, and we don't see any need to comment the issue at all.

d) It's not good for our business to say bad things about one of our partners.

It's always a problem for a company when there are some concerns about their partners. You have commitments with your partners (NVIDIA, ATI, AMD, Intel, …) and with everyone who use or buy 3dmark. Interests of your partners and interests of your users are different. How do you deal with that ?

The whole shader optimization issue could have gone much smoother if (well, if there wouldn't have been any tension between us and Nvidia, but that's beside the point) the gfx IHVs would have first discussed the issue with us and then implemented the shader replacements. We were asked to do some scrambling of the shader instructions, in order to find out who does direct shader identification and replacement + this was at the time considered an all bad thing. Further discussions have shown, that under some circumstances (that we're trying to define at the moment) shader replacements can be accepted. If these discussions would have been held first, and shader replacement would have been implemented in drivers only afterwards, I doubt there would have been problems like the ones we ran into. Also, the whole shader replacement thing was a brand new issue. Nobody knew at the time how to react to it. This is a fine example of why it is important that we don't impulsively comment all issues immediately, rather first consider them carefully. And this is why our response might take some time, but it does not always mean that we don't give a d---. It does mean that sometime though

I fully agree.

But now, with all these issues, Futuremark and 3dmark has unfortunately lost a part of its "credibility". Do you plan to do something to get it back?

I think that it could be very easy for you

1. Let IHV use good optimisations. -> You must say clearly what you'll consider as a good optimisation and do everything to avoid the use of bad optimisations.
2. Add an optional path in 3dmark with no optimisations and ask IHV to respect this path.
3. Add the possibility to switch from standard precision to partial precision.
4. Add the fly mode in the public version. It's really easy to use optimisations which look good only in some circumstances. The fly mode is an easy way to check for optimisations problems.

It'll be very easy for you to do this with the standard or the professional version of 3dmark03. Why don't you do something like that?


I have another question :

What's the main objective of 3dmark03: show how a card will run current and future games or show the 3d abilities of a graphic card?

The difference between these 2 objectives is small but very important. For example : I think that everyone knows that the floating point power of Radeon are higher than floating point power of GeForce FX. DX9 shader are floating point shader. Games developers will have to use complex DX9 shader not too much because a big part of the market won't have the ability to run them at a sufficient speed. Do you think that a benchmark has to do the same thing?

 

[Reverend]

I am not breaking any NDAs by saying the following!

Futuremark is still studying what is meant by "optimizations" (i.e. everything that this incredibly important word means). Beyond3D is also studying it. We are all also studying if such "optimizations" should be app-specific or application-wide (generally, but even "generally" can be an important word).

I think we can all gather that FM, as well as the entire industry, is still learning. That's a Good Thing... hopefully, the results will be enlightening... as well as determinant (in many regards).

 

[swanlee]

Application specific optimizations cannot be a good thing in a benchmark. Because if they are truely application specific and what you are running is a benchmark the only reason for these optimizations is to bump up the score. The only acceptable optimization that would effect a benchmark would be one that effects other games in the same way, but then it would not be an application specific optimization it would either be an api optimization or possibly a game engine optimization.

I think either way you cut it Nvidia is trying to force it's will on 3dmark and 3dmark are folding. If nividia's hardware was better it would follow dx9 specs and not have so much toruble with 3dmark and then Nvidia would not have to optimize a freakin benchmark to get decent scores.

This all boils down to nvidia's newest hardware is not up to snuff and Nvidia cannot accept 2nd place so they must push there weight around to change the way benchmarks are done so they look better.

 

[Malo]

We are all also studying if such "optimizations" should be app-specific or application-wide (generally, but even "generally" can be an important word).

rev nvidia themselves stated on those slides that these "optimisations" (which will forever be in quotations) must accelerate more than just a specific application. I really don't understand how FM can allow such an app specific hack when nvidia promise otherwise.....

 

[micron]

swanlee,
[sarcastic]Thanks for enlightening us.[/sarcastic]

 

[jb]

Patric Ojala,

I am sadden that you let any optimizations in. You folks made it very clear in your first public statement and the 330 patch. That sets a precedent. Then you back off these statements and now are chaging your word to allow optimizations in. This is very bad. A benchmark that is sythentic should never allow any optimizations of any kind. You ask the GPU to do a certian amount of work, in a certian way in a given time. Buy sticking to a very firm stanace you create a controlled enviroment that very usefull. However now one vendor is compling with your first statement while the other is not. The fact that you let this stand adds bias to your benchmark weather you like it or not

 

[digitalwanderer]

Thanks Rev, I keep forgetting you folks are active with FM behind the scenes.

I'll reserve judgement until some more info is known....if B3D suddenly pulls out of FM I'll consider that pretty telling even if no one can say a thing due to NDAs.

 

[Tridam]

Actually, some DX9 shader optimizations are good because of the lack of a runtime compiler in DX9. ATI can optimize for co-issue and NVIDIA can optimize for register usage. These ones are good optimizations.

 

[WaltC]

Patric,

I just read your post in your personal forum here at B3d, and I've read your comments in this thread as well. The following was among the most troubling of your remarks, and the most disappointing to me personally:

...Immediately as Nvidia left our development program, getting them back into the program was raised to a very high priority. Our highest priority naturally is and remains the development of high quality and impartial benchmarks.

I'm trying to get you readers to understand what a great thing it is for us that Nvidia re-joined our development program. I think this is the best thing our company has achieved since we launched 3DMark03. The Futuremark employees went through some tough times back there, but we're all happy to get Nvidia back, including AJ (even though some weird rumours tell differently) .Immediately as Nvidia left our development program, getting them back into the program was raised to a very high priority. Our highest priority naturally is and remains the development of high quality and impartial benchmarks.

I have to tell you I find this statement simply baffling. nVidia belonged to the 3dMk03 development program for at least 15 months out of the 18-month 3dMK03 development cycle (as they didn't quit the program until December of '02.) What was it, precisely, that you could obtain from nVidia by "getting them back" into the program that would allow you to make 3dMK03 a "high quality and impartial" benchmark that you failed to obtain from nVidia during the long months in which it belonged to the FM program during the 3dMK03 development cycle? I simply cannot imagine what that might be. Please elaborate.

nVidia was in the 3dMK03 development cycle long enough to learn how to cheat the benchmark with dispatch when it shipped--it boggles the mind how you might think that getting nVidia back had anything at all to do with making 3dMK03 a "high quality and impartial" benchmark, since presumably that's what it was when it shipped.

How is nVidia being in or out of the program relevant to 3dMk03 being a high-quality benchmark as you define it, since nVidia was a full partner for virtually the entire development cycle of 3dMK03?

Hopefully, you are not declaring that 3dMK03 when it shipped was anything but a high-quality and impartial benchmark...? This is baffling for me, honestly.

I'm trying to get you readers to understand what a great thing it is for us that Nvidia re-joined our development program. I think this is the best thing our company has achieved since we launched 3DMark03. The Futuremark employees went through some tough times back there, but we're all happy to get Nvidia back, including AJ (even though some weird rumours tell differently) .

It should be obvious why Nvidia is so important for our development program, but I'll repeat some of the key reasons for those unfamiliar with our company's business.

In order to produce top quality forward looking benchmarks, we need the input and cooperation of all the major players of the industry.

So, how is it you did not receive "input and cooperation" from nVidia for 3dMK03 development when nVidia was paying you for the privilege of providing such throughout the 3dMK03 development cycle? I'm sure I don't have to remind you that the events which occurred prior to nVidia leaving your program, prior to 3dMk03 shipping, prior to nVidia cheating the benchmark, and prior to your audit report detailing exactly how they cheated it--prior to these things nVidia was just as much a part of the FM program as it is today.

Your supposition seems to be that you require nVidia's participation to be able to write a good benchmark, and yet you had nVidia's participation when you wrote 3dMK03 and before *nVidia* made the elective decision to pull out and discredit your company's software.

So if nVidia was unable to assist you in writing a high-quality, impartial benchmark prior to resigning from the program, what has changed so that now nVidia's participation is *required* to produce that result?

How could we otherwise get the highly confidential information of where the industry is going next? Each 3DMark version is aimed at the new if not next generation of hardware, and it is of crushial importance to have comprehensive information of that next generation, in order to make a benchmark for it in advance. In addition to making just high quality benchmarks, they must be impartial, and it is not convincing to release an impartial benchmark developed in cooperation with just one of the major IHVs. If we would not have got Nvidia back, we would have still done our best to optimize it also with Nvidia's upcoming products in mind, but it would have been way harder.

Again, this does nothing to explain the puzzling issue you've raised, which is why you failed to obtain that "comprehensive information of that next generation, in order to make a benchmark for it in advance" from nVidia during the development of the 3dMK03 software (since nVidia belonged to the program for the bulk of the development cycle.) Of course, the truth is you did obtain it, but that nVidia wasn't happy with what you did with the "comprehensive information" and so it quit your program. That's my take on it based on what you've said.

Of course you've glossed over the very odd structure of your business model--which is that companies pay you for the privilege of providing their own next-gen information to you so that ostensibly that information will be used in the construction of a benchmark which will portray their products in a positive light and help them sell those products. Hopefully, you can see the glaring conflict of interest in doing that and in providing a "high-quality, impartial" benchmark for the 3d-card buying public. To that end I can see nVidia's point in quitting--their view was that they were paying you for specific services which you were no longer providing them. It's the conflict of interest inherent in your business model which has caused all of this to happen, IMO.

Sadly, I must regrettably conclude that nVidia's participation is a requirement for your company because of nVidia's financial contribution to your company based on the membership fees it pays you. Obviously, I guess you've agreed to start giving nVidia its money's worth and nVidia is now back in. Seriously, unless you guys start doing some serious "rethinking" about your business model and understand the inherent conflict of interest within it I don't see much of a future for you in this endeavor, and I see more of this same sort of trouble for you on the horizon.

 

[micron]

ouch.

 

[Mark]

@Futuremark guys (worm, Patric, and whoever else).

I paid for 3DMark03 because at the time it provided honest and meaningful results for my videocard reviews. Now, after the deals made with nVidia to allow optimizations for the benchmark, those results are difficult to understand and compare. The fact is 3DMark03 simply can't be trusted to provide those honest results anymore. Therefore, through Futuremarks own actions (and inactions), the 3Dmark03 program is now worthless and my money has been wasted.

 

[digitalwanderer]

@Futuremark guys (worm, Patric, and whoever else).

I paid for 3DMark03 because at the time it provided honest and meaningful results for my videocard reviews. Now, after the deals made with nVidia to allow optimizations for the benchmark, those results are difficult to understand and compare. The fact is 3DMark03 simply can't be trusted to provide those honest results anymore. Therefore, through Futuremarks own actions (and inactions), the 3Dmark03 program is now worthless and my money has been wasted.

What I'm interested in seeing is how the FM people react to such a post, because there are a whole bunch of similar minded folk out there right now who are waiting for an answer/some reassurance and it's another case of the silence being all deafening.

Well, 'cept for nVidia who keeps floating rumors... :?

Seriously, does FM plan to react to this or address it in some way soon?

 

[micron]

Whats interesting is that on the Futuremark forums, there were members there who owned 5900Ultra's, and were'nt aloud to submit scores. I'm talking about people who purchased the pro version, and couldnt even use it to it's intended functionality. Some were requesting to get their money back, it never happened. What a tough situation for Futuremark......

 

[digitalwanderer]

Whats interesting is that on the Futuremark forums, there were members there who owned 5900Ultra's, and were'nt aloud to submit scores. I'm talking about people who purchased the pro version, and couldnt even use it to it's intended functionality. Some were requesting to get their money back, it never happened. What a tough situation for Futuremark......

Not FMs fault, blame that on nVidia. FM's policy on allowed submissions was quite clear, and the drivers in question did not qualify.

 

[micron]

Whats interesting is that on the Futuremark forums, there were members there who owned 5900Ultra's, and were'nt aloud to submit scores. I'm talking about people who purchased the pro version, and couldnt even use it to it's intended functionality. Some were requesting to get their money back, it never happened. What a tough situation for Futuremark......

Not FMs fault, blame that on nVidia. FM's policy on allowed submissions was quite clear, and the drivers in question did not qualify.

FM needs to put a clause in their 'Pro' purchase contract that warns of score submision refusals if any of the IHV's become cheating bastards.
I dont think any of the guy's who couldnt submit scores saw this coming.
I woulda been pissed if I was them...
It's Futuremark's problem.
(edit)
I understand what you are saying , I'm simply trying to put myself in the position of the people who were not aloud to submit scores.

 

[Malo]

It's Futuremark's problem.

it's FM's problem that the drivers didn't meet the standards required for a fair and comparable benchmark rating? please explain....