Very informative, interesting and well-written, I agree. That doesn't change my view on the situation, though. What FM seems to say (feel free to disagree) is that Nvidia did not "cheat" per se, but apply optimizations that would be legal and welcomed for other software (games, for example), but are not for benchmarks.
As of now, the only thing that would restore my faith in FM would be if they issued a technical (as opposed to PR) paper explaining for each and every "former cheat" of both ATI and Nvidia why this "former cheat" is indeed a "slight optimization", as formulated by their PR person on ExtremeTech. I must say I would be pretty interested on a technical answer on the clipping planes, for example, and how said driver-level clipping planes taking advantage of know camera paths would be useful for gaming purposes.
As for FM considering including vendor-specific path in their code, why not, provided :
- the vendor, which would have to be a beta member, proposes a specific code path to FM. Driver-level stuff is a no-no, and any detection of driver-level stuff gets all scores of said driver set removed from the ORB, and an email fired to any person having posted such a score ("We are sorry, but the manufacturer of your video card chipset
cheated, which is why we removed your score. Contact your video card chipset's manufacturers for further details").
- FM independantly analyzes the output of every proposed code path, which has to be a "generic inside the app" optimization (ie for a shader, the vendor-specific shader has to be a mathematically equivalent of the generic shader proposed by FM). This would allow for both instructions reorganization and partial precision to play a role, provided they produce the exact same output.
- all path-specific proposals are presented for all beta members to see and comment upon.
- FM has the final say about what makes it to the code and what doesn't
- the benchmark would propose both "use vendor path" and "use generic path" options in all its configuration, including the free downloadable version.
There is no way we can spare resources to check each released driver for optimizations.
Which is in direct contradiction with the PDF explaining why synthetic benchmarks are better, and can be interpreted as providing another escape way to close your eyes on future
cheats by any company.
3DMark scores are only comparable if drivers perform exactly the work 3DMark instructs them to do.
We know that. But since you say that you won't investigage all future driver releases, how are we supposed to know that those scores are not comparable in the first place ? Do you realize that saying such a thing without dedicating yourselves to an extensive research of future
cheats is indeed invalidating 3DM ? Both statements can't go together.
Many people have speculated that Nvidia would have paid us to publish the statement.
Actually, many people speculated that Nvidia would have either paid you or threatened you with legal action. I'm curious as to why you don't comment on the second part...
Even though it might feel unfair and wrong for us to approach Nvidia after all this arguing, this is vital for the continuity of our benchmark products.
Let me respectfully disagree here, Patric. Getting better with NV is vital for the
commercial future of FM, either because of all the smearing they did (directly through company statements or through their "black ops PR specialists", the "guys with webpages"), or because of legal threats, no doubt about it. But "continuity" relies on much more than commercial future, it also relies on public acceptance of the product as an useful tool. I am not against synthetic benchmarks, I think they are useful tools to use in addition to in-game benchmarks, but I consider the specific software 3DMark2003 will be of zero value once ATI or NV release new drivers, especially if those give a large performance increase. This is not because of some (actual or pretended) opposition to synthetic benchmarks, but because FM destroyed their own credibility by calling blatant cheats (those clipping planes are still stuck in my throat) "slight optimizations". This indicates a desire to have a better relationship with a major IHV that goes way beyond any pretention at objectivity.
Call a cat a cat, a cheat a cheat, and you will restore this credibility.
I've even read some pretty absurd benchmark recommendations by real professional reviewers
We want names !!! Actually, if you refer to the various "guys with webpages", they hardly qualify as "professional reviewers" in the first place...
