How benchmark 3D Cards, today

Discussion in 'Graphics and Semiconductor Industry' started by Griffith, Mar 3, 2006.

  1. RedBlackDevil

    Banned

    Joined:
    Mar 4, 2006
    Messages:
    44
    Likes Received:
    0
    How make usefull posts, today :roll:
     
  2. Moloch

    Moloch God of Wicked Games
    Veteran

    Joined:
    Jun 20, 2002
    Messages:
    2,981
    Likes Received:
    72
    Ya.. how to stick to topic today.
    me- pick 5 or so of the latest (good) games which stress modern hardware and have some nice shadermark 2.1 benchmarks and some other synthetic benchmarks (not 3dmark 06).
    Then check out IQ on a perframe basic for atleast 1 hour of game play.. maybe they only cheat on certain frames :D
    er.. I mean play some games on the card carefully exampling IQ at default settings and with all opts disabled.
    A nice texture filtering app would be good also, like what beyond3d uses.
     
  3. RedBlackDevil

    Banned

    Joined:
    Mar 4, 2006
    Messages:
    44
    Likes Received:
    0
    I think that a reviewer have to ask himself

    "what the consumer want from a review?"

    I believe that a consumer want this things:

    a) what 3d hardware is better with actual game?
    -> what settings make me play with MINIMUM framerate above 30 fps with common heavy-demand games?

    b) what 3d hardware is better to play with eye-candy mode?
    -> I don't care about costs, I want to know whic card give me the best IQ with all settings maxxed

    c) what 3d hardware will mantein his value in future?
    -> I care barely of actual games, I'm buying a new pc and I will stay with this card for at least 2 year, please benchmark it with worse situations possibile, so I can compare to others card and make my choice

    when I talk about 3d hardware, I talk about 3d hardware, no software tricks, no drivers 'optimizations', no cheats
    the same condition for all the cards tested.

    another point is that, if ATI R520-R580 have no hit, from 4xMSAA / 8xAF to 6xMSAA / 16xAF is unfair don't benchark this, only because with nvidia any AA over 4x will produce an incredible hit
    if a card has a value, a good point, the reviewer have to underline this, in order to make a good service to his reader


    at now is still possibile do this, jumping from a review site to another, but no one yet give all the info required from consumers
    my best fair, detailed prefered bench-review sites are hardware-fr.com and beyond3d.com
     
  4. _xxx_

    Banned

    Joined:
    Aug 3, 2004
    Messages:
    5,008
    Likes Received:
    86
    Location:
    Stuttgart, Germany
  5. Mendel

    Mendel Mr. Upgrade
    Veteran

    Joined:
    Nov 28, 2003
    Messages:
    1,350
    Likes Received:
    17
    Location:
    Finland

    Okay, here's how I would test graphics cards: I'd make it a double blind test, as follows:

    6 Graphics cards (2 value cards, 2 mainstream cards, 2 high end cards, 1 each from ati, 1 from Nvidia)
    D6 dice
    1 piece of paper, envelope and a pencil
    6 assistants to test the cards, each selecting a number from 1 to 6
    6 identical computers (other than gpus) or a whole lot more of time and one computer
    1 assistant to roll the dice, each card is labelled with a number.
    1 assistant for changing the settings.

    -The dice rolling assistant will not know numbers of the 6 assistants (who is who). He will only know which assistant number got which card, the assistant will never meet the others to tell them which card they got. Instead the results of dice rolling will be written at this time to an envelope which gets sealed.

    -The assistants testing the cards will never know which card they got. They will not be allowed to touch the control panels so they won't even know if they had an nvidia card or an Ati card.

    -Settings changing assistant will not meet anyone of the testers, the tester assistants will leave the room during settings change and return once complete and the settings changer has left the room.

    The testers will continue testing the card until they have reached their conclusion about it, write notes, and then will increase their number by one, therefore getting a new card to test. In other words, every tester will test every card with a number of games and settings, not knowing which card they used at which time.

    In the end, after all testers have tested all cards and formed their conclusions about it, an article will be written to a website, the writer of the article only knows the cards and assistants by number.

    After that, the dice rolling assistant will give the writer of the article the envelope containing information about which card was which and that information will be added to the end of the article or throughout the article for clarity.

    What do you think? Quite an undertaking, no? No bias at least!
     
    #85 Mendel, Mar 7, 2006
    Last edited by a moderator: Mar 7, 2006
  6. _xxx_

    Banned

    Joined:
    Aug 3, 2004
    Messages:
    5,008
    Likes Received:
    86
    Location:
    Stuttgart, Germany
    I'm perfectly happy with the current testing procedures combined over various sites. When you check out the reviews on B3D, [H], Anand, Hexus, ixbt and Guru3D you'll get a good enough picture. Obviosly, not so from a single review on a single site.

    I wish people (especially various f-boys) would just stop bitching, that would make it much better already.

    EDIT: and you'll definitely NEVER be able to satisfy everyone, so why even bother?
     
  7. Grestorn

    Newcomer

    Joined:
    Jun 7, 2004
    Messages:
    62
    Likes Received:
    2
    Well, the assistant who is to change the settings has to be competent and unbiased, since he would be able to manipulate the results.

    Other than that: Sounds great. Now you just need the manpower and time to do it :)
     
  8. Druga Runda

    Druga Runda Sleepy Substitute
    Regular

    Joined:
    Sep 30, 2003
    Messages:
    647
    Likes Received:
    43
    if noone bothered there would be no progress, good to see people trying to get better [​IMG]
     
  9. RedBlackDevil

    Banned

    Joined:
    Mar 4, 2006
    Messages:
    44
    Likes Received:
    0
  10. _xxx_

    Banned

    Joined:
    Aug 3, 2004
    Messages:
    5,008
    Likes Received:
    86
    Location:
    Stuttgart, Germany
    No hostility here, where do you get that from?

    We compare cards for speed in a buttload of games/apps, we do compare them for IQ. What more can/will you do? It takes no Einstein to figure out that there are not many more options left.

    EDIT LOL@ "3-4 years old benchmark style". You DO know what the word benchmark means?
     
  11. HawkeyeGnu

    Newcomer

    Joined:
    Sep 29, 2005
    Messages:
    17
    Likes Received:
    1
    Location:
    UK
    *UK grammar nazi mode*

    Nope, "practise" & "practised" are the verb forms of "practice" in REAL English :twisted:

    Sorry for OT!
     
  12. Geo

    Geo Mostly Harmless
    Legend

    Joined:
    Apr 22, 2002
    Messages:
    9,116
    Likes Received:
    213
    Location:
    Uffda-land
    I would think that's fairly obvious --because a signficiant portion of the community is unhappy with the status quo! Now, whether they can ever be made happy, or at least if they can be made happy without pissing off a different portion of the community, is an entirely different question! :razz:
     
  13. _xxx_

    Banned

    Joined:
    Aug 3, 2004
    Messages:
    5,008
    Likes Received:
    86
    Location:
    Stuttgart, Germany
    But again the question: what options do you have for benchmarking besides comparing speed over a (huge) variety of games/apps and IQ-comparisons?

    I can't think of any, maybe a comparison of "warm, fuzzy feeling"-ness or the colours of cards, dunno :razz:
     
  14. Geo

    Geo Mostly Harmless
    Legend

    Joined:
    Apr 22, 2002
    Messages:
    9,116
    Likes Received:
    213
    Location:
    Uffda-land
    I'd like to see a range of cpus in *every* review. For every guy who's itching to have a new gpu and has an FX60 to pair with it, there are five or more guys who are itching for a new gpu but have something significantly less exalted to pair it with.

    I roll my eyes every time I see a mid-range or value gpu paired with the latest/greatest cpu for benching. It's not that that isn't interesting, as one data point, but it's almost certainly a hell of a lot less interesting to the potential market that card is aimed at than something more mid-rangy, and even a cpu that is bordering on "oldy/moldy" status!

    Did I just suggest creating a shit-load more work for the reviewers? Did I just piss off IHV PR with the idea of showing their new babies in "less than optimum" platforms? Yeah, well, cry me a river! :razz: If the subject is what would actually be useful in improving reviews for, y'know, real consumers, then there ya go.
     
  15. _xxx_

    Banned

    Joined:
    Aug 3, 2004
    Messages:
    5,008
    Likes Received:
    86
    Location:
    Stuttgart, Germany
    CPU-variety is a very good point indeed.

    EDIT: didn't [H] and Anand test with one high-end and one lower-end CPU some 2-3 years ago?
     
    #95 _xxx_, Mar 7, 2006
    Last edited by a moderator: Mar 7, 2006
  16. FrgMstr

    Newcomer

    Joined:
    Jun 26, 2002
    Messages:
    223
    Likes Received:
    4
    Location:
    Lucas, TX
    We have a CPU / Video Card scaling article that is nearing completeion now. 6 different CPUs, 3 from Intel, 3 from AMD. Dual core included in both. 6 or 7 video cards used. Only used NVIDIA. BFGTech sponsored the article. We stuck with one brand as it is my opnion that it will give us results that are easier to represent without a bunch of argument on the video card end. We have used real-world gameplay and IQ scaling as the measure for comparison as we do in current video card evaluations.

    This has been a tremendous amount of work and required a lot of time and money to get done.
     
  17. _xxx_

    Banned

    Joined:
    Aug 3, 2004
    Messages:
    5,008
    Likes Received:
    86
    Location:
    Stuttgart, Germany
    But I can imagine you can put 2-3 "standard" systems together with low, mid and high-end CPU respectively for video card tests? You'd only have to change them 'bout twice a year.

    Right now, a sytem with AthlonXP 2500+, one with Athlon64 3800+ and one with the current top CPU would do the job fine methinks. So the budget people drooling over the new GFX-card would know if it makes sense putting it into his system.

    I know it would be a bit of work, but it would be nice.

    Kyle, didn't you do something like that when you tested Doom3 back then?

    EDIT: the search function on [H] is truly abysmal, there's some potential for serious improvements there ;)
     
    #97 _xxx_, Mar 7, 2006
    Last edited by a moderator: Mar 7, 2006
  18. Geo

    Geo Mostly Harmless
    Legend

    Joined:
    Apr 22, 2002
    Messages:
    9,116
    Likes Received:
    213
    Location:
    Uffda-land
    I don't doubt it. Resource realites are resource realities. Doesn't change the answer to the question tho.

    Probably a periodic visiting, such as the article you're pointing at, is a reasonable compromise. It'd be nice to see more sites doing it more often. If several sites were doing such once a year it'd probably mostly cover the need. Tho on "release day" there is still the danger that old Joe is liable to succumb to his itch and run off to Newegg for a gpu that isn't going to improve his quality of gameplay as much as he's thinking it will due to the cpu he has to play it with.
     
  19. Altcon

    Regular

    Joined:
    Sep 19, 2003
    Messages:
    357
    Likes Received:
    1
    Location:
    side A
    Sounds Fun!!!
    You should tr tp pitch it as a hit collector...it night even spurr a whole new breed of review method using blind and double blind tests!
    Any takers?
     
  20. Geo

    Geo Mostly Harmless
    Legend

    Joined:
    Apr 22, 2002
    Messages:
    9,116
    Likes Received:
    213
    Location:
    Uffda-land
    I'd settle for *one* game in the current suite (tho preferably the most demanding) tested that way on each new gpu review. Provide at least a quick thumbnail for Joe Enthusiast to ponder before he plonks down for a new gpu to go with an older cpu/mobo.

    Tho I also think it's pretty silly to be testing mid-range/value gpus with the current top cpu anyway. I might even reverse the above --test it for the entire suite with something more mid-rangy cpu-wise, and provide one further test with the most demanding game with the burliest cpu you can get your hands on.
     
Loading...

Share This Page

  • About Us

    Beyond3D has been around for over a decade and prides itself on being the best place on the web for in-depth, technically-driven discussion and analysis of 3D graphics hardware. If you love pixels and transistors, you've come to the right place!

    Beyond3D is proudly published by GPU Tools Ltd.
Loading...