[H] 3D Testing Methodology Discussion

Discussion in '3D Hardware, Software & Output Devices' started by Arty, Jan 28, 2008.

Thread Status:
Not open for further replies.
  1. FrgMstr

    Newcomer

    Joined:
    Jun 26, 2002
    Messages:
    223
    Likes Received:
    4
    Location:
    Lucas, TX
    Sorry, I did not understand you point. Brent and I are just normal guys that like game, the same as you. Just somewhere it got crazy and turned into a job. That is a double edged sword.
     
  2. digitalwanderer

    digitalwanderer Dangerously Mirthful
    Legend

    Joined:
    Feb 19, 2002
    Messages:
    18,452
    Likes Received:
    3,020
    Location:
    Winfield, IN USA
    Yeah, I realized/had that pointed out to me later. I wrote it before I'd fully woke up from a puppy nap, but then decided to let it stand since I got this thing about trying to edit history.

    My bad, my apologies. Sometimes when I try to be funny it's funny for the wrong reasons, like I make meself into a bit of an asshat. It was funny in my head when I wrote it, honest. :oops:
     
  3. Squilliam

    Squilliam Beyond3d isn't defined yet
    Veteran

    Joined:
    Jan 11, 2008
    Messages:
    3,495
    Likes Received:
    114
    Location:
    New Zealand
    Since we have your attention Kyle. Do you ever go back to old reviews and update them with errata etc, if you find significant enough differences in future driver updates for the same game? Also, has this ever changed the conclusion of a past review?
     
  4. FrgMstr

    Newcomer

    Joined:
    Jun 26, 2002
    Messages:
    223
    Likes Received:
    4
    Location:
    Lucas, TX
    No. In the video card arena we genenerally do enough reviews of retail cards to keep the performance conclusions up to date. Sort of one of those things that is ongoing till the card drops so far in price that we don't really worry about it anymore. A good example of a hugely successful card later in its life would be the X1900 series.

    (And by our metrics, once a card is below US$175, our readers will no longer read the content. We tried hitting below that price point earlier last year and the pages went unread, so we backed away from "cheap" cards.)

    http://www.hardocp.com/article.html?art=MTE5OCwxLCxoZW50aHVzaWFzdA==

    That x1900 series card got better the older it got IMO. Drivers matured and it represented a great value at its price point.
     
  5. Sound_Card

    Regular

    Joined:
    Nov 24, 2006
    Messages:
    936
    Likes Received:
    4
    Location:
    San Antonio, TX
  6. Mark

    Mark aka Ratchet
    Regular

    Joined:
    Apr 12, 2002
    Messages:
    604
    Likes Received:
    33
    Location:
    Newfoundland, Canada
    Now that's a shitty chart scale.

    Anyway, I'll mention this again since the [H] guys have yet to clarify why they say my approach to benchmarking is wrong.

    I verify, using fraps, that my custom timedemos spit out results that are very close to what I can expect from a regular "real-world" play session. Using these timedemos with automatic benchmark scripts allows me to run peformance tests over a lot of different graphics cards, games, resolutions, and settings (AA/AF combos). These results are objective and extremely accurate, and there are enough of them there so that a user can pick and choose how he wants to compare the cards based on his own hardware (say he only cares about 1680x1050, his monitors native res. There are results there for him).

    [H] does real-world gameplay, like me, but the process is so time consuming they can't present results from a wide variety of games, hardware, resolutions, or variations of settings. Not only that, but these results that are prone to inaccuracies simply because they are gathered manually (basic human errors).

    According to [H] my approach is wrong. Why?
     
  7. digitalwanderer

    digitalwanderer Dangerously Mirthful
    Legend

    Joined:
    Feb 19, 2002
    Messages:
    18,452
    Likes Received:
    3,020
    Location:
    Winfield, IN USA
    Because it's not real world gaming, silly. :roll:
     
  8. Bouncing Zabaglione Bros.

    Legend

    Joined:
    Jun 24, 2003
    Messages:
    6,363
    Likes Received:
    83

    [H] would say because you don't know what optimisations are in place when you do a playback (for instance, would AI be disabled if the game's playback system knows you're playing back a demo?), and you don't know if the graphic drivers detect the playback state of the game and enable cheats.
     
  9. Squilliam

    Squilliam Beyond3d isn't defined yet
    Veteran

    Joined:
    Jan 11, 2008
    Messages:
    3,495
    Likes Received:
    114
    Location:
    New Zealand
    The first one, AI would even out comparisons as graphics cards are tested on the best CPU's anyway/ :twisted: (Seriously if you guys can't find anywhere to put your ex reviewed stuff, well... I could... Help you out there. :wink:

    Actually if you changed the name of the exe to something else and repeated the tests. As long as the differences remain constant you know that special sauce hasn't been added. Actually in the case of driver cheats on time demos, the reviewers' experience does come into play a lot there.
     
  10. digitalwanderer

    digitalwanderer Dangerously Mirthful
    Legend

    Joined:
    Feb 19, 2002
    Messages:
    18,452
    Likes Received:
    3,020
    Location:
    Winfield, IN USA
    Changing the .exe can fuck up legitimate work-arounds and such though lately I've found.
     
  11. Mark

    Mark aka Ratchet
    Regular

    Joined:
    Apr 12, 2002
    Messages:
    604
    Likes Received:
    33
    Location:
    Newfoundland, Canada
    remember that the output of the timedemo compares nicely to the results from manual run-throughs with fraps
     
  12. Mark

    Mark aka Ratchet
    Regular

    Joined:
    Apr 12, 2002
    Messages:
    604
    Likes Received:
    33
    Location:
    Newfoundland, Canada
    bump for kyle/brent in hopes that they'll respond
    [post]1126096[/post]
     
  13. FrgMstr

    Newcomer

    Joined:
    Jun 26, 2002
    Messages:
    223
    Likes Received:
    4
    Location:
    Lucas, TX
    Prove it?
     
  14. Mark

    Mark aka Ratchet
    Regular

    Joined:
    Apr 12, 2002
    Messages:
    604
    Likes Received:
    33
    Location:
    Newfoundland, Canada
    How would you prove your method?
     
  15. Mark

    Mark aka Ratchet
    Regular

    Joined:
    Apr 12, 2002
    Messages:
    604
    Likes Received:
    33
    Location:
    Newfoundland, Canada
    I guess actually proving your method is a difficult thing. You're pretty much relying on your reputation as an honest and objective reviewer when you describe how you do your tests.

    As I've said. I record long timedemos, very long. Sometimes of entire levels, and if the timedemo results don't match with the results that Fraps gives during a similar play through then I rerecord a new timedemo until the results do match. Then I rerun different settings and resolutions manually and compare with the timedemos again and again to make sure that the results will always match. This is time consuming, of course, but I only have to do it once until I get an acceptable timedemo for the game then I can use that timedemo for all my reviews. For games that have some variety in environments I often generate and run serveral timedemos, then average the results. Or even run several timedemos several times. There are even a few games that aren't 100% reliable where I will run 3 different timedemos 3 different times per resolution/setting (9 times total, just to get 1 result) just to make sure that the results are accurate.

    I firmly believe that this is a very accurate and objective way to benchmark graphics cards, so I take offense when someone tells me repeatedly that it is invalid and wrong without giving me any explanations as to why.
     
  16. FrgMstr

    Newcomer

    Joined:
    Jun 26, 2002
    Messages:
    223
    Likes Received:
    4
    Location:
    Lucas, TX
    HardOCP does not use a "method." We evaluate real gameplay and base our conclusions on that.
     
  17. FrgMstr

    Newcomer

    Joined:
    Jun 26, 2002
    Messages:
    223
    Likes Received:
    4
    Location:
    Lucas, TX
    Well I guess if you "believe" it must be fact? Since you are offended, why don't you just prove me wrong? I asked you to prove it above, and you totally dodged the question instead of answering it.
     
  18. Tim Murray

    Tim Murray the Windom Earle of mobile SOCs
    Veteran

    Joined:
    May 25, 2003
    Messages:
    3,278
    Likes Received:
    66
    Location:
    Mountain View, CA
    Prove that your results are meaningful at all (e.g., for future applications) except in the context of performance in games that are available when you went to review! No one has yet to convince me that performance in a current game is a meaningful indicator of the performance of a single component in the machine, yet that's all that [H] gives me.

    Also, I'm getting totally mixed messages from you:
    Yet you continually claim that everyone who is not you or Brent is somehow doing their job incorrectly, producing invalid results, and then being totally oblivious of this fact. But if you're regular guys with no special training, why are you the arbiters of what is and is not correct compared to guys with backgrounds in computer science, computer engineering or 3D programming, who know how to use all of the performance measuring tools given to them (via PIX, PerfKit, PerfStudio, looking at actual shaders, etc.)?
     
  19. Mark

    Mark aka Ratchet
    Regular

    Joined:
    Apr 12, 2002
    Messages:
    604
    Likes Received:
    33
    Location:
    Newfoundland, Canada
    Talk about dodging questions. I asked you 3 times in this thread how my method is wrong and you haven't given me an explanation yet.

    Wait, prove you wrong about what? Where did I say your method was wrong. I asked you to show me how my method was wrong. You've been saying the [H] way is right while everyone else is wrong... shouldn't the onus be on you?

    No, scratch that question. The onus is on you.

    You've been dragging the names of everyone whose ever done a graphics review through the mud these last few years, if you can't give a reasonable explanation as to why my method is wrong (sure seems like you can't) then maybe you should stop going off about it every chance you get and show a little respect to other reviewers.
     
  20. silent_guy

    Veteran Subscriber

    Joined:
    Mar 7, 2006
    Messages:
    3,754
    Likes Received:
    1,380
    It's a really effective, isn't it?
    1. Adopt a dogma: "Existing Techniques are Bad."
    2. Develop a new technique that can't be falsified.
    3. Claim that your new technique is better than the others...
    4. (Profit?)

    Since there's no easy to be prove your technique wrong, you'll always win... or at least sow doubt in the minds of those who don't know better.

    It works even better if you actually truly believe what you're selling.

    (They must have read the manual about Promoting Intelligent Design really well.)

    It's not that there's no merit to reporting out how a game feels in real life, but the simple fact that the obtained result are not repeatable make them inherently unscientific and open to valid criticism.
     
Loading...
Thread Status:
Not open for further replies.

Share This Page

  • About Us

    Beyond3D has been around for over a decade and prides itself on being the best place on the web for in-depth, technically-driven discussion and analysis of 3D graphics hardware. If you love pixels and transistors, you've come to the right place!

    Beyond3D is proudly published by GPU Tools Ltd.
Loading...