On the Splinter Cell benchmark

[Brent]

I think they should use 2 demos one custom private and one custom public and use the private one as a control so to speak.
Or make a set of criteria for creating demos to ensure that demos are generally created using the same type of rendering stressors. Then you can use the review demo once and release it at the time of review. Then create a new one for each review. the only thing that would be questionable would be the strength of the demo creation "rules" and how well a creator stuck to them.

edit
also in regards to the first point maybe a reviewer can just use a private demo only and release it to the public at some arbitrary point in time (i.e. six monthes after creation) at which point the cycle repeats.

i like that idea, a control

 

[indio]

For any scientifc study to be accepted , using some sort of control group is the de facto standard. In a perfect world the control would be a separate reviewer on separate hardware without the knowledge of the other tester. However budgets and time constraints make it near impossible.
A control timedemo or replay would be useful to show any obvious fudging. If something seems outside the reviewers preordained margin of error one could investigate further. You could even use 2 controls . You wouldn't need to run the whole review one or two extra times . I would just take a sampling and determine is it needs further analysis.
We're not looking for absolute precision here (no pun intended)
on fps numbers. Just a margin of error. Your just looking for a reason to investigate further. If there's a problem there are many ways to handle it , from not publishing the results to calling an attorney.