Are all timedemos now invalid because of this 3DMark Fiasco?

[ben6]

Re: Bigger problem.

I think the biggest problem with the 3DMark 2003 situation isn't that we can find ways to get around it but that it changes the nature of reviewing.

(For the record I am a reviewer myself and write from this perspective).
'
One of the most important functions of journalists (online and otherwise) is to act as watchdogs and to blow the whistle when we detect corporate abuse, cheating, or other improper / unethical behavior. Investigative reporting and detective work is often a part of doing our job correctly and that's as it should be.

Benchmarks are used to report on hardware performance and oftentimes to detect potential issues or problems, but the NVIDIA problem and this current discussion casts a great deal of doubt on just how much we can trust the tools we use. Unlike the ATI Quack / Quake hack which was easily detected by any user, NVIDIA's cheating was much more subtle and could've gone detected a long time if not for inquisitiveness on the part of ExtremeTech.

My point is simple: Reviewers should not have to spend our time testing and re-testing to ensure the very tools we are attempting to use remain valid, especially when the invalidity has been introduced by a third party. Companies like having their products used as benchmarks; its an immense amount of free exposure. They won't take kindly to performance hijacking by third parties or to having their products abused and twisted for the marketing purposes of another company.

The bottom line? This type of cheating only hurts the entire industry. NVIDIA's Detonator drivers have always been famed for their ability ot increase performance. NExt time they release a new set people are going to be constantly asking whether or not the drivers cheat. Meanwhile we as reviewers have to question the performance of every benchmark we run and wonder if they've been similarly hijacked, potentially going so far as to develop new testing tools just to ensure our old ones are still valid.

We shouldn't have to do that. This type of hijacking isn't fair to readers or consumers, and its not fair to us. If we can't trust manufacturers to maintain some level of fair play then no review is going to be worth the paper its printed on simply because no one will ever know if the numbers are telling the truth.

Or could it be that the methodology that most reviewers have used and still use to this day needs to be changed?

There's stilll way too many reviews that don't bother to look at anything but introduction, basic features, benchmarks , conclusion.

Benchmarking has it's place, don't get me wrong, but I've felt for a long long time that the way 90% of the reviewers do reviews is just wrong.

Working together with software makers, IHVs, other websites to have new tools in games is a start, playing the games you play with a framecounter like FRAPs is another part of the story, but more can be done.