B3D custom demos not availed to public -- Agree?

[WaltC]

I've changed my mind. I think B3D should release whatever benchmarks they use.

Why? Think of all the incorrect benchmarks that have been posted to major sites in just the last couple weeks or so:

• 
  
  
  That was my initial reaction as well, but on reflection I think the idea sort of misses the point. B3d is rightly concerned with steps B3d can take to ensure that the hardware guys can't cheat its benchmarks. IE, whatever B3d does isn't going to change the behavior at other web sites--they're going to keep on doing what they are already doing, regardless. So B3d's primary focus here should be on itself and keeping the benchmarks private guarantees no cheating--releasing it to the public guarantees an opposite probability (it seems to me.)
  
  Let's look at Anand's 1600x1200 x4 FSAA blunder. It's obviously an error. However, the error is made most apparent by the published findings of other web sites (not to mention common sense) which show a much different result. Even though Q3 is certainly not private, people at home do not have 5900U's to run to validate or dispute any such published results--so the result is very similar to Anand running a private benchmark anyway.
  
  There's just no perfect solution, unfortunately. Even if a particular web site states that it is keeping its benchmarks private and out of the hands of the hardware guys, we still won't know whether that is the truth. So it becomes a matter of who you trust. But if benches are released publicly then the possibility of hardware companies "optimizing" for them becomes a distinct probability. As we move forward I think that "trustworthiness" will become a major commodity within the Internet community. Sites which have it will propser and those which don't, won't.

 

[AzBat]

I totally agree with you. Security through obscurity isnt going to work

I have to disagree with that. Again, I see it as a "good thing" to not publically release a home recorded demo, as long as you also test the "typical prevailing" demos for a reference.

Which is exactly what I suggested to Dave privately. More work is a bitch I know, but what other options are there? Ever since I started benchmarking(96 or so) I always felt that the more data points you have the better. Eventually the data will point out mistakes and patterns of performance and features.

I really feel for the benchmark testing guys. Hopefully newer tools will make it easier to determine who is cheating and who is not. This could lead to another idea I suggested to Dave, The League of Extraordinary Benchmarkers. But seriously it's similar in function to the Code of Ethics mentioned previously. I just think benchmarkers are going to need to organize themselves. NVIDIA and ATI would be less apt to cheat if they knew an independent group of benchmarkers were actively searching for cheating.

Tommy McClain