Proper Mantle testing in muliplayer games.

Discussion in '3D Hardware, Software & Output Devices' started by DavidGraham, Jun 6, 2014.

  1. DavidGraham

    Veteran

    Joined:
    Dec 22, 2009
    Messages:
    2,749
    Likes Received:
    2,518
    We have discussed this before in the Mantle thread, but I was baffled none of the famous sites actually considered a proper testing methodology for BF4 mantle/dx comparisons. Until an obscure arabian site actually found and did it:



    A video of the testing procedure:
    https://www.youtube.com/watch?v=JPJICrRkpWM

    Link to the rest of the benchmarks:
    http://www.araboverclockers.net/directx-to-reign-supreme-once-again/2/

    I hope this becomes the standard way of testing things from now on.
     
  2. 3dilettante

    Legend Alpha

    Joined:
    Sep 15, 2003
    Messages:
    8,122
    Likes Received:
    2,873
    Location:
    Well within 3d
    I hope to look at this more in-depth later, but spectator mode support in various games and how closely it matches the performance behavior of direct play has been spotty at times.
     
  3. DavidGraham

    Veteran

    Joined:
    Dec 22, 2009
    Messages:
    2,749
    Likes Received:
    2,518
    True, but given the other options, this does seem like the most accurate. It is like a timedemo, or a built-in benchmark.
     
  4. Ryan Smith

    Regular

    Joined:
    Mar 26, 2010
    Messages:
    609
    Likes Received:
    1,036
    Location:
    PCIe x16_1
    Just speaking solely for myself here, it is something we had considered because it does have merit. But ultimately it's something we declined for a couple of reasons.

    1) I don't have access to two identical testbeds. Actually there's a hair bit of concern about the cards being non-identical due to differing leakage characteristics (a bigger problem on the ref 290s than the better cooled non-ref), but mostly the lack of a second identical testbed.

    2) It's non-repeatable in as much as you can't do more than 2 cards like that, and you can't repeat it in the future (say when AMD updates their drivers).

    That said, 3dilettante's concerns aside, it's a respectable way of doing one-off testing.:smile:
     
  5. fuboi

    Newcomer

    Joined:
    Aug 6, 2011
    Messages:
    90
    Likes Received:
    45
    Their lack of frame distribution statistics makes it almost useless. Instantaneous minimum frame rate is useless. Percentage of frames below 60fps would be more interesting. In the youtube video you can see one of the machines hitting sub-60fps quite a bit. I hope one of the frame statistics sites does a test as a proof of concept. Testing with switched APIs on both machines (Test A: Machine A DX, Machine B Mantle, Test B: swap APIs) and comparing distributions ratios should give a better reference point to remove leakage differences.
     
  6. DavidGraham

    Veteran

    Joined:
    Dec 22, 2009
    Messages:
    2,749
    Likes Received:
    2,518
    Yeah, it's an expensive test due to this requirement alone. increasing the number of cards tested will balloon the hardware requirements to an insane level!

    At least you won't have to worry about that if you are comparing NV vs AMD.

    Agreed.
     
  7. silent_guy

    Veteran Subscriber

    Joined:
    Mar 7, 2006
    Messages:
    3,754
    Likes Received:
    1,379
    (OT, but I'm a bit surprised that even a site like Anandtech doesn't have identical testbeds to parallelize benchmarks. These kind of setups aren't that outrageously expensive?)
     
  8. Ryan Smith

    Regular

    Joined:
    Mar 26, 2010
    Messages:
    609
    Likes Received:
    1,036
    Location:
    PCIe x16_1
    We're not in a single office, so we don't have the advantage of shared hardware.
     
Loading...

Share This Page

  • About Us

    Beyond3D has been around for over a decade and prides itself on being the best place on the web for in-depth, technically-driven discussion and analysis of 3D graphics hardware. If you love pixels and transistors, you've come to the right place!

    Beyond3D is proudly published by GPU Tools Ltd.
Loading...