jvd said:
what i want to know is whats going to happen with the industry . New test shows that nvidia has cheated in almost ever benchmark there is . Next thing we know they are cheating in gl doom 1 . So if we can't even trust benchmarks how will we know what card is better .
That's the job of the competent hardware reviewer...
It has been the result of competent people, not incompetent, who have revealed the current crop of information about nVidia that many people do think is relevent enough to skew their purchasing decisions.
Benchmarks should never suffer driver hacks which seek to modify the default benchmark workload, because benchmarks are used for comparative judgements in hardware performance and simply are invalid if one IHV's drivers cut the workload to garner better performance numbers. That is a gross misrepresentation and a deceit and needs and begs to be called out. That is cheating in the purest sense of the word.
Benchmarks also should not be optimized for, even though optimization isn't cheating, unless the IHV does similar application-specific optimizations for > 50% of all the 3D titles that ship under a certain API, for which frame-rate performance numbers are important. Even though optimization deals with ordering the workload for more efficiency as opposed to dishonestly hacking it down to create false performance impressions, it is important that 3D benchmarks accurately portray an average-case performance for the IHV's hardware in 51% or greater of the 3D software shipped under the API the benchmark addresses. This ensures that the benchmark will provide useful information.
But games--games--now, to me they are an entirely different proposition. The purpose of games is to that they be played and not used as benchmarks (obviously.) Off hand, I'd say we don't have to worry about drivers which "cut the workload" for games in general because drivers which do so will enjoy a high probability of being so buggy in actual game play that they would be useless. So I don't think we can extend the definition of cheating in benchmarks to cheating in games, since on one hand the purposes for the two kinds of software are vastly different, and on the other it is likely impossible to cheat a game in the same way you would cheat a benchmark. Games are where the practice of optimization can shine--or not--depending on what the IHV brings to the table.
So, does it matter what nVidia or ATi do in their drivers relative to the D3 game engine? I don't think so. If one or the other company is cutting corners by lowering precision for better frame rates, for instance, the competent hardware reviewer will catch this--or at least should catch it. In glaring instances the general public may well catch it as well. At that point consumers can make decisions congruent with ther own preferences as to the sort of environment they'd prefer to have. But I wouldn't call this kind of thing "cheating" in the same way I'd call cutting down a benchmark's workload to achieve spurious benchmark numbers "cheating." If, for instance, one company proclaims or implies, "We run Doom3 in fp32" but an analysis reveals that the company is actually running the game at a lower precision, I wouldn't really call that "cheating", either. Heh--it's "lying", of course, but still a different matter...
This is the kind of thing observant consumers and hardware reviewers will eventually discover, and so consumers can make choices accordingly.
But, as far as games go, I think a distinction is critically important: if a hardware reviewer says, "Today we are going to present you with testing information on X-product running UT2K3," but what he does instead is provide a few bar charts featuring the performance of the Antulus Fly-By timedemo, he has not done as he stated he would do. IE, commercial timedemos based on game engines are not "games" they are in effect "benchmarks" and must be treated as benchmarks and not games (since an IHV can code his driver to cheat a commercially available timedemo as well as a commercially available benchmark.)
So with the stipulation that game engine-based commercial timedemos are not games themselves but rather are benchmarks, I'd say that the distinction between games and benchmarks involving these matters is an important one. Optimization in games can be a good thing, but cheating and/or special-casing benchmarks is not, IMO.
