Neutral Benchmarks

[Reverend]

Aki, thanks for your participation and your post. Markus told me he never has time to participate so you are indeed welcomed here!

You make many valid points but I think the biggest consideration here is about the focus and expertise of websites/media/press, not the validity of using certain benchmarking apps or games for hardware reviews. It is sometimes difficult to know what you have learned after reading a hardware review ("It gives us faster FPS, end of story" vs "It gives us faster FPS because if you'll look at the pixel fillrate graph you can see that the difference is due to the number of pixel pipes", as examples). It is probably even more difficult to know what you, as a reviewer, think the public should know. To many reviewers, 3DMark is a tool to use to arrive at a "conclusion" about a piece of hardware and not necessarily for analyzing a piece of hardware. To some others, 3DMark is a tool for analyzing the hardware. More often than not, such different "sets" of reviewers are a mutually exclusive bunch.

PS. Sorry if I'm mendering off topic a little but I think the above is the crux of the "matter".

 

[Pete]

Could you "use" any more quotes, Rev?

 

[Reverend]

"I" "was" "just" "trying" "to" "be" "diplomatic", "Pete".

 

[Reverend]

I just wanted to add that Aki's post is pretty much a longer version of what Markus wrote me a long time ago. The point was that unless you know exactly what a game is doing when you benchmark it, you never know if certain things are happening... like automatically scaling down textures with different video cards, dropping pixel shader effects with different cards... basically scaling automatically according to features, CPU, video memory etc. Serious Sam is probably the most famous example.

Some sites have evidently followed NVIDIA's position regarding 3DMark03 ("right, games is always the better ones to use" they think) without their own investigation. That is not carrying responsibility.

 

[AJ]

Aki, thanks for your participation and your post. Markus told me he never has time to participate so you are indeed welcomed here!

You make many valid points but I think the biggest consideration here is about the focus and expertise of websites/media/press, not the validity of using certain benchmarking apps or games for hardware reviews. It is sometimes difficult to know what you have learned after reading a hardware review ("It gives us faster FPS, end of story" vs "It gives us faster FPS because if you'll look at the pixel fillrate graph you can see that the difference is due to the number of pixel pipes", as examples). It is probably even more difficult to know what you, as a reviewer, think the public should know. To many reviewers, 3DMark is a tool to use to arrive at a "conclusion" about a piece of hardware and not necessarily for analyzing a piece of hardware. To some others, 3DMark is a tool for analyzing the hardware. More often than not, such different "sets" of reviewers are a mutually exclusive bunch.

PS. Sorry if I'm mendering off topic a little but I think the above is the crux of the "matter".

Hi Rev,

Good point about the target groups. I'll try to open up that a little. In addition to reviewers there are also the end users (Joe 'but I do have a computer' Sixpack), and manufacturers to consider. Building a benchmark that would fully and completely satisfy all requests would be impossible because these target groups have contradicting needs. However we do try to improve and incorporate new features with each release.

A couple of good examples that were requests from press and reviewers were the frame based mode and image quality tests. I think the IQ part doesn't need further explanation, but years we've been demanded to give a 'Quake like' option for running 3DMark. Internally we've had a lot of debates about how we should tackle the question about 'time based' vs. 'frame based' mode of measuring performance. Given how many professional reviewers requested this we did implement this to 3DMark03.

That's just one example, but I hope it shows that while we can't include all suggestions for improvement to every release (because frankly many of them tend to be contradicting requests) we do try to incorporate as many as we can.

Cheers,

AJ