[H] 3D Testing Methodology Discussion

Will do.

I have never claimed that. You are lying, or making things up as many forum goers here do. Don't put your words in my mouth. If you are to question me about something I have said, I would suggest you do some homework and get a quote instead of dreaming up baseless "claims." You are obviously confused about the facts. Sounds like you are making a lot of stuff up.

However, to try and answer your somewhat fragmented question, I guess if being a scientist or engineer makes gives you 10 years of experience in qualifying the gaming experience supplied by hardware, then they would likely be better at it. I know Perfkit is best gaming evaluation tool I have ever seen. Yes, that is sarcasm. I think that is what you are were going for in your question, but it failed miserably.

 

I find it funny to think that I am even familiar with "your method." I don't read your content. I also find it funny that you think my general comments are pointed at you and your method. So to directly answer you, I have no idea what you really even do here besides moderate this forum. I have not read an article on B3D in years. Sorry.

I guess if you are worried about my actions you should do something about it. Prove me wrong or even better, prove yourself right. That is something I WOULD read on your content pages. Better yet, stop reading HardOCP.com. I would say it seems you keep coming back for more and if you don't like it, then why do it.

I think it is interesting that you say the burden of proof is on me. You want me to prove that our evaluations of the gameplay provided by a video card is superior to reviews of video cards done with timedemo tools? I can do that. Article will be up Monday. Seems to be pretty clear to me if you have half a brain, but I can prove it.

Now that is settled how bout you go about proving "your method" (whatever it may be) on the graphically challenging title gamers' hands right now? Prove it to me with some game that matters, not some 2 year old crap engine that does not stress the hardware. Because quite frankly, the "method" you described earlier sounded like you doing nothing but fishing for results that finally match, and I would hardly call that objective, or scientific, or however you put it. Disregard the results that do not prop up your own and look for ones that do. Yeah, that's the ticket.

 

I'm looking forward to that article Monday and am reserving all me snarky comments until then.

 

Now I don't want to put words in anyone's mouths, but I think what he's trying to say is that having an understanding of the underlying hardware and graphics algorithms at work is what allows one to generalize results across different games and even predict future results. Indeed if you take the purely phenomenological approach of "I rate game X on hardware Y at experience index Z" without trying to explain the result, then you result is only useful when evaluating game X on hardware Y. Thus unless you're planning to test every game on all pieces of hardware, the result is trivially less useful than one to which some deductive reasoning has been applied.

PerfKit - like many other developer tools - is designed to answer exactly the above question: "*why* is my application slow"? Once you've determined that there's a performance problem (or not) for game X on hardware Y, and you want to generalize that result, it's good to know things like where the bottleneck is. With such information one can predict whether the performance problem is localized to - say - the specific rendering path used in a game (for instance, deferred or forward rendering) - or rather a systemic problem that is likely to resurface in other games or in future games.

Forgive me if I sound condescending; that is not my intention. However as a self-proclaimed gamer I'm not sure how much of the graphics algorithm/development side you're familiar with and thus I figured it might be useful to inject some relevant information into this conversation.

To summarize, PerfKit and related tools can certainly give you good data about application performance characteristics if you know how to interpret the information. This sort of lower-level analysis is critical to being able to generalize performance results, and thus is something that should be of interest to anyone writing graphics hardware reviews.

 

I stopped reading hardocp years ago to be frank, and I never came to this thread looking for anything other than an explanation as to why you continue to dump on anyone who uses timedemos in reviews. I've already told you why I think using timedemos has advantages over your methods. I guess I'll have to wait till Monday to get your explanation why you think otherwise.

FWIW, many new games can benchmarked using timedemos or other automated or semi-automated procedures if you take the time to figure them out, and I've never used games that weren't considered modern or unpopular at the time.

 

I stopped doing that a long time ago. Really Kyle. People in glass houses shouldn't throw stones. You criticize people for the way they do their reviews. But do not seem to be willing to accept any criticism yourself. I am really surprised the staff at beyond3d lets a thread like this go on. They certainly don't have to prove themselves to you.

 

AndyTX explained exactly what I meant better than I could. Thanks for that.

Should anyone want to continue discussing this with me, they're welcome to take it to PM.

 

Indeed, given where this is going I think it's better for all parties involved to take this to PM if they want to - thread closed. If you disagree, it turns out (shock, horror) that I also have a PM box. Who would have thought?