An interesting take on the performance duel at ExtemeTech

Discussion in '3D Hardware, Software & Output Devices' started by cthellis42, Aug 20, 2003.

  1. cthellis42

    cthellis42 Hoopy Frood
    Legend

    Joined:
    Jun 15, 2003
    Messages:
    5,890
    Likes Received:
    33
    Location:
    Out of my gourd
    I rather like the roundabout method as hey... don't we do things the "normal" way all the time anyway? :)
     
  2. Joe DeFuria

    Legend

    Joined:
    Feb 6, 2002
    Messages:
    5,994
    Likes Received:
    70
    I have been waiting a long time for someone to take this approach to comparisons. I am very happy to see it being pursued.

    But sadly, there are two GAPING holes which makes this poor execution of a good idea:

    1)....despite the emphasis on quality....there is not a single screenshot comparison that allows the readers to compare for themselves what the relative quality is at the given settings.

    2) ET appears to take the approach of "if both cards are at 1280x960, with 4X AA and 8x aniso, that means the same image quality." Nowhere in that Article is there even any hint that the same "settings" on different cards yields different quality results. That is a huge, glaring omission which does nothing but mislead the readers.

    I mean "Commanche 4" for example, is called a "draw" because each card performs at the 16x12, @ 30 FPS target with 4X AA and 8X aniso. No acknowledgement that ATI's 4X AA is vastly superior to nVidia's 4X AA. And god knows what "Aniso" settings were used...for either card.

    Sigh...
     
  3. nelg

    Veteran

    Joined:
    Jan 26, 2003
    Messages:
    1,557
    Likes Received:
    42
    Location:
    Toronto
    Well at least someone listened. :D

    Joe, your are absolutely correct though. It is not enough to simply say that such and such is the sweet spot for this card and game. The experience must be compared with the other card.
     
  4. Ante P

    Veteran

    Joined:
    Mar 24, 2002
    Messages:
    1,448
    Likes Received:
    0
    What kind of monstrous monitor do you have where you can see a difference between two 4x MSAA implementations at 1600x1200?

    (No I'm not trying to start a fight, it's a real question.)
     
  5. jb

    jb
    Veteran

    Joined:
    Feb 6, 2002
    Messages:
    1,636
    Likes Received:
    7
    Look at poles in racing games or things like that and you can see a difference at 16x12..

    that pipe demo would be a good test if you can run it that high of a res.
     
  6. Joe DeFuria

    Legend

    Joined:
    Feb 6, 2002
    Messages:
    5,994
    Likes Received:
    70
    You mean, between one MSAA solution that is sparse gird and gamma corrected, and one MSAA solution that is ordered grid and not gamma corrected?

    Probably any monitor that supports 1600x1200.
     
  7. Ante P

    Veteran

    Joined:
    Mar 24, 2002
    Messages:
    1,448
    Likes Received:
    0
    I sure don't on my 19". At least not in any of the games I ever play. (I'm certainly not trying to imply that you don't though so don't get me wrong here.)

    I can't see the difference between ATis 4x and 6x at that res either.

    Perhaps I just have bad eyes or an unusually small 19". :)

    But I do agree that screenshots should have been made. The article was pretty pointless without them.

    4x AA is just a setting. They treated it like it's an absolute degree of image quality.
    I mean how would they have treated the 2x AA mode of a Xabre which doesn't even take two samples. Or the 16x D3D mode on a FX which only takes 12 samples etc. etc. etc.

    I like the general idea behind the article though.
    It certainly saves a lot of space in a review too, no more wading through 50 uninteresting test results, just straight to the point.
     
  8. Malo

    Malo Yak Mechanicum
    Legend Veteran Subscriber

    Joined:
    Feb 9, 2002
    Messages:
    7,447
    Likes Received:
    3,468
    Location:
    Pennsylvania

    wtf is with this? Surely there must be some driver issue or rendering implementation to cause such a problem on the ATI? Anyone guess why FS2004 would kill the 9800Pro?
     
  9. WaltC

    Veteran

    Joined:
    Jul 22, 2002
    Messages:
    2,710
    Likes Received:
    8
    Location:
    BelleVue Sanatorium, Billary, NY. Patient privile
    You guys who say you like this approach have got me baffled...;)

    What does this:

    Have to do with the "Quality of 3d" in Jedi Knight II? First of all, the basic IQ settings aren't the same, secondly there's nary a screenshot in sight.

    Stranger than strange--after listing the above results the author then says at the end of the article that the "3d Quality" between the above products in the above game compared at the above settings was a "draw."

    I'm afraid I've missed the point entirely somehow.

    You know--want to artificially constrain frame rates? Here's a novel suggestion: turn on vsync and set the refresh rate to correspond to the cap you want on frame rates. That way you don't have to run one card at a higher IQ setting just to load the vpu to drop down to close to the frame-rate target you want. But anyway, there was no attempt to justify any of the findings that I could see.

    Look at the difference in the IQ settings between the cards. What does "hitting an arbitrary frame rate cap" have to do with 3d Image Quality between two cards running different IQ settings...?

    OK, what's wrong with comparing 4x FSAA/8x AF for the R9800P with 4x FSAA/8xAF on the GFFX, and 6x/16x to 8x/8x, respectively? You want to artificially cap frame rates--do it with vsync.

    Again, what's wrong with comparing R9800P @ 4xAA/8xAF to GFFX at 4xAA/8xAF & 6x/16x to 8x/8x, respectively?

    I mean, what exactly is the purpose of talking about capping frame rates in the first place when the subject of the article is 3d image quality? What does frame rate have to do with it at all? Seems to me you'd want to, if anything, leave out any mention of frame rate and concentrate on screen shots, instead.

    And, in many cases the frame rate "targets" were completely missed anyway (not suprising considering the strange method of attempting to hit them.) Again, why not cap frame rates with vsync and refresh rate, compare the cards at equal settings for IQ with screen shots, and then make judgements?

    I guess I don't get what this article had to do with "3D Graphics: Quality Matters" at all...where does "quality" enter into it?
     
  10. Joe DeFuria

    Legend

    Joined:
    Feb 6, 2002
    Messages:
    5,994
    Likes Received:
    70
    Walt,

    I'll try and explain. :)

    The premise is, "beyond a certain frame rate, performance doesn't matter."

    This isn't really arguable...although choosing which frame rate is the magic "it doesn't matter past this point" is certainly debatable. The ExtremeTech author's do acknowledge this, and they offered their "magic frame rates", dependent on the genre.

    FPS Shooters: 60 FPS.
    Flight Sims: 30 FPS.

    (etc.)

    That's reasonable, but one can certainly choose something else...or choose a "minimum" FPS, vs. an average, or whatnot.

    So basically, the goal is to "crank up the image quality settings as high as you can, but still at least maintain that magic frame rate".

    The Jedi Knight II case is pathologic: no matter HOW HIGH they cranked the settings, they couldn't get the FPS down to 60. ;)

    So, "both cards" are fast enough...but the problem is, they don't tell you how they qualitatively compare!

    The thing is...they are not artificially constraining frame rates. They are increasing image quality to hit a target.

    Heh...the whole point is to precisely determine which card can "withstand" a higher IQ setting and still hit the target! :)

    In other words, the card with the highest IQ that still hits the target...is the winner!

    I think this is a completely valid approach. Again, it was just executed miserably...since they make blatantly gross (and wrong) assumptions about image quality....namely: all AA and aniso settings between cards are equivalent in image quality. (Let alone other tweaks like LOD bias, color precision and overall fidelity are completely ignored.)

    They also don't give any screenshots like "THIS is what Card A will deliver at 30 FPS....and THIS is what Card B will deliver at 30 FP. You decide for yourself which one looks better."
     
  11. cthellis42

    cthellis42 Hoopy Frood
    Legend

    Joined:
    Jun 15, 2003
    Messages:
    5,890
    Likes Received:
    33
    Location:
    Out of my gourd
    Hey, I mainly like DIFFERENT approaches for being different! It's interesting, just flawed. As Joe says, we're not given a solid grounding in true IQ comparisons, so we don't know how things really stack up. If they'd prefaced that exact performance review with an in-depth look at the AA/AF methods--chock full of the right kinds of screenshots we've all seen before--I think many people would find it very handy overall.

    On a side note, is it just me, or did the the review NEVER mention if it was a 128 or 256MB version of the 9800 Pro?
     
  12. MikeC

    Newcomer

    Joined:
    Feb 9, 2002
    Messages:
    194
    Likes Received:
    0
    What about giving the reader the following information from FRAPS, which I just finished compiling after logging six hours of multiplayer game performance in BF1942 over the past three days?


    BERLIN MAP


    1920x1440 - NO AA - NO AF

    2003-08-18 20:02:59 - BF1942
    Frames: 11288 - Time: 212686ms - Avg: 53.073 - Min: 32 - Max: 79

    2003-08-18 20:06:35 - BF1942
    Frames: 5348 - Time: 96919ms - Avg: 55.180 - Min: 41 - Max: 85


    1600x1200 - 0X AA - 8X AF

    2003-08-17 08:42:24 - bf1942
    Frames: 21753 - Time: 398934ms - Avg: 54.527 - Min: 30 - Max: 148

    2003-08-19 20:58:37 - BF1942
    Frames: 12491 - Time: 232224ms - Avg: 53.788 - Min: 30 - Max: 239

    2003-08-19 21:02:55 - BF1942
    Frames: 15252 - Time: 263378ms - Avg: 57.909 - Min: 26 - Max: 195

    2003-08-19 21:07:30 - BF1942
    Frames: 13203 - Time: 242188ms - Avg: 54.515 - Min: 29 - Max: 195


    1024x768 - 4X AA - 8X AF

    2003-08-17 22:35:09 - bf1942
    Frames: 18939 - Time: 234837ms - Avg: 80.647 - Min: 59 - Max: 139

    2003-08-17 22:45:48 - bf1942
    Frames: 40550 - Time: 469435ms - Avg: 86.380 - Min: 46 - Max: 159

    2003-08-17 22:53:39 - bf1942
    Frames: 57539 - Time: 763878ms - Avg: 75.324 - Min: 46 - Max: 137

    2003-08-17 23:06:27 - bf1942
    Frames: 19883 - Time: 230131ms - Avg: 86.398 - Min: 44 - Max: 158

    2003-08-17 23:10:19 - bf1942
    Frames: 18639 - Time: 199196ms - Avg: 93.571 - Min: 61 - Max: 202


    1280x960 - 4X AA - 8X AF

    2003-08-18 17:58:57 - bf1942
    Frames: 19273 - Time: 378804ms - Avg: 50.878 - Min: 27 - Max: 99

    2003-08-18 18:08:50 - bf1942
    Frames: 26680 - Time: 442095ms - Avg: 60.349 - Min: 31 - Max: 104

    2003-08-18 18:21:02 - bf1942
    Frames: 35722 - Time: 601385ms - Avg: 59.399 - Min: 37 - Max: 108

    2003-08-18 18:33:04 - bf1942
    Frames: 23757 - Time: 421046ms - Avg: 56.423 - Min: 35 - Max: 110

    2003-08-18 18:40:07 - bf1942
    Frames: 25522 - Time: 429708ms - Avg: 59.393 - Min: 31 - Max: 115


    1024x768 - 6X AA - 8X AF

    2003-08-18 19:04:45 - bf1942
    Frames: 17834 - Time: 306240ms - Avg: 58.235 - Min: 35 - Max: 100

    2003-08-18 19:31:09 - bf1942
    Frames: 20105 - Time: 299831ms - Avg: 67.054 - Min: 39 - Max: 100

    2003-08-18 19:45:03 - bf1942
    Frames: 25507 - Time: 370713ms - Avg: 68.805 - Min: 26 - Max: 140

    2003-08-18 19:51:46 - bf1942
    Frames: 15459 - Time: 226276ms - Avg: 68.319 - Min: 32 - Max: 123


    ABERDEEN MAP

    1024x768 - 6X AA - 8X AF

    2003-08-19 22:42:44 - bf1942
    Frames: 37272 - Time: 275166ms - Avg: 135.452 - Min: 58 - Max: 231


    WAKE ISLAND MAP

    1024x768 - 6X AA - 8X AF

    2003-08-19 22:49:19 - bf1942
    Frames: 90980 - Time: 724852ms - Avg: 125.515 - Min: 64 - Max: 289

    2003-08-19 23:01:28 - bf1942
    Frames: 75990 - Time: 805187ms - Avg: 94.375 - Min: 28 - Max: 268

    2003-08-19 23:14:56 - bf1942
    Frames: 20644 - Time: 213637ms - Avg: 96.631 - Min: 56 - Max: 174
     
  13. digitalwanderer

    digitalwanderer Dangerously Mirthful
    Legend

    Joined:
    Feb 19, 2002
    Messages:
    17,469
    Likes Received:
    1,923
    Location:
    Winfield, IN USA
    Y'know if you could make that into a pretty graph it would really be nice!

    Hmmm.....time to go play with making some graphs. Figuring out a good fraps template seems very worthwhile. Thanks Mike. :)
     
  14. Joe DeFuria

    Legend

    Joined:
    Feb 6, 2002
    Messages:
    5,994
    Likes Received:
    70
    He didn't mention it, but the links in the article to get the current prices correspond to the 128 MB version...
     
  15. Bolloxoid

    Newcomer

    Joined:
    May 15, 2003
    Messages:
    191
    Likes Received:
    0
    I think the approach used in this review makes sense as an alternative way of examining graphics cards because this is what end-users actually do when selecting in-game settings, ie. find the best settings the card can cope with while offering satisfactory performance.

    What is missing are (as already mentioned) screenshots, but also qualitative impressions about the level of performance. Enabling certain options in certain games may cause annoying stuttering and periodic dips in frame rate even when the average fps seems to be high. These kinds of things should definitely be taken into account before declaring the performace to be satisfactory.

    About the frame rate goals, they are always debatable but I think an average frame rate of 30 fps is not acceptable, because it means that the actual frame rate is often below that.

    But hey, this is a first review of its kind so we can't expect it to be satisfactory in all respects. It's a good thing people occasionally try to do things differently.
     
  16. WaltC

    Veteran

    Joined:
    Jul 22, 2002
    Messages:
    2,710
    Likes Received:
    8
    Location:
    BelleVue Sanatorium, Billary, NY. Patient privile

    I agree these are "reasonable" expectations for frame rates, but I can't see any association with "quality." I would also say as regards the "quality" of playability that maximum frame-rate targets are essentially meaningless--what counts I would think would be the minimum frame-rate averages for games--ie, as long as you hit the minimum average frame rates necessary for smooth gameplay the importance of higher frame rates beyond that is nil.

    So, it seems to me he got it backwards...;) Rather than artifically attempt to lower the frame rates by upping the IQ settings, one would would simply set the IQ settings one wants and simply ignore all frame-rates above the minimum required averages. The point would be to decide what level of IQ you want to see--first--and then check to see if the products hit your minimum frame-rate average targets at those IQ levels--ignoring all frame rates above those minimum requirements (as they would have nothing to do with "quality" in any sense.)

    I think I understand what his intent was--I just think he went about it bass ackwards...;)

    Yes--exactly. There is no correlation between what they did and 3d graphics quality--which is the problem with the approach, I think.

    But in that case the article is flawed such that it appears the author himself did not understand what he was trying to do. In Jedi Knight II the R9800 came closest to the target with the highest degree of IQ setting between the cards, yet the author declares the cards a "draw." I mean, Joe, this indicates a difference between your interpretation of "the winner" and the author's, right?...;)


    Is it any secret to anyone that you can reduce a card's frame rate by upping its IQ settings for AF and FSAA?...;) Conversely, should anyone be surprised that you can increase frame rates by doing the reverse? On one card he's upping the IQ settings but on the other he's lowering them, simply to try to hit an arbitrary frame rate cap that he thinks has value, all of which tells us absolutely nothing about "3d quality" between the two products in those instances...;) That's about as artificial as things get, IMO.

    Also, the methodology is the reverse of what people do with 3d cards. The first thing they do is to set the IQ settings they want and then play the game to see if the minimum framerate performance they require is supported. If it's not, only then will they lower IQ settings--they also might simply lower resolution alone and leave FSAA and AF settings unchanged. If they find they have plenty of power at their initial IQ settings they may well up them, provided frame-rate performance remains suitable for them.

    I don't think I've ever seen a case where an end user says first thing, "This is a flight sim and I want 30 fps, no more and no less, and I'm going to sit here and tinker with my IQ settings until I can get it." Ditto, fpshooters @ 60 fps. People generally see about IQ first and only start adjusting it downwards if they find that the card won't hit the minimum frame rates they need for playability. I've never heard of anyone being concerned about capping his framerates to a less than optimum condition regardless of the IQ settings used...;)


    Agree mostly, except I think it would be a valid approach only if minimum frame-rate averages were the target, with anything over the minimums simply ignored. I just don't think that capping framerates arbitrarily and then *comparing* cards at different IQ settings simply to hit the arbitrary framerate cap, if possible, has much to tell us about "quality" in 3d...;)

    I will say this, if the article had been entitled: "How to use your 3d card's IQ settings to cap your framerates to <60 frames per second, Sometimes" I probably would still have thought it bizarre, but wouldn't have argued with it....;)
     
  17. Reverend

    Banned

    Joined:
    Jan 31, 2002
    Messages:
    3,266
    Likes Received:
    24
    Perhaps the main point of this article by Dave Salvator wasn't to "equalize" IQ. Perhaps he felt that the IQ differences between the two cards (AA, AF, whatnots) have been well documented so far and that the purpose of this article was to simply see what settings each card can be set to to give almost similar FPS in various games.

    Or something like that... I'm feeling a little "lost" right now, having written the entire review of a GFFX5900 in a single day (and there's my day job too!).
     
  18. Joe DeFuria

    Legend

    Joined:
    Feb 6, 2002
    Messages:
    5,994
    Likes Received:
    70
    I'll just have to respectfully disagree with you here. :)

    IMO, your method really isn't different than what we get today: reviewers "set a quality level, and then run benchmarks."

    Indeed. :) Very bad execution. R9800 should have been delcared the winner.

    No, no one should be surprised at all, and the article is not presenting this as if we should be.

    Well, the issue I see, is what I explained earlier...he makes assumptions that 4XAA = 4XAA, for example.

    I mean, we all know that "higher resolution is better". So all else being equal, 1600x1200 is better than 1024x768. That's self-evident. No one thinks that "ATIs implementation of 1600x1200 can be different than nVidia's 1600x1200" nor should we. And I think he just carried that logic over to other "imgae quality tweaks" like AA and aniso. Thie higher the number, the better....and 4X from one vender = 4X from another. We know that's not right, of course, but IMO, either he doesn't know this, or he doesn't see any difference.

    Atually, I disagree completely. When I play games, I push the image quality as high as I can go while maintaining a "good feel." That's exactly the process he's trying to do.

    But you're really saying the same thing I am. If the minimum performance is being met though...then they'll try and increase image quality further, as long as the frame rate is met.

    That's more or less what you just described. Certainly, the don't sit down with FRAPs and see if they can get "30 FPS", but I know I do play around with image quiality...raising and lowering, until I get the maximum image quality for the performance I want.

    That's another valid approach sure. But you might actually find that "minimums" aren't all that GPU dependent (might be system hitches, swapping, etc.).

    It's not arbitrary. It's at point where HE finds it acceptable. Personally, I would have probably chosen 60FPs for ANY game, and cloer to 100 FPS for twitch games, but everyone is different.
     
  19. AzBat

    AzBat Agent of the Bat
    Legend Veteran

    Joined:
    Apr 1, 2002
    Messages:
    6,206
    Likes Received:
    2,086
    Location:
    Alma, AR
    I would like to make a comment that Futuremark, early in 3DMark's life, contemplated doing something like this. That is, set a frame rate target and then keep increasing image quality or polygon counts until it exceeds the target. The score would either be the maximum amount of polygons it could render at the target or the maximum amount of quality. Considering that performance is getting closer and closer among competitors, it might be a good time to start adding something like this to the next 3DMark. They have already done something similar by introducing the frame-based rendering tests for 3DMark03. Maybe they could take it a step further?

    Tommy McClain
     
  20. Quitch

    Veteran

    Joined:
    Jun 11, 2003
    Messages:
    1,521
    Likes Received:
    4
    Location:
    UK
    I fail to see what this approach brings to the table. I can already do this with existing reviews, if they involve any depth or wide range of testing (how hard is it to check all the tables to see when each card was above 60 FPS?). Not only can I see whereabouts the card hit my "magic number", I can see what they can do around it too, making me a much more informed consumer than if someone else is deciding what FPS I want.

    The approach taken in this review gives you almost no leeway for additional tweaking, since they have already hit the supposed "magic number". In a traditional review I could still see whereabouts I would be able to take the card, while giving myself space for LOD tweaks, full trilinear, etc.
     
Loading...

Share This Page

  • About Us

    Beyond3D has been around for over a decade and prides itself on being the best place on the web for in-depth, technically-driven discussion and analysis of 3D graphics hardware. If you love pixels and transistors, you've come to the right place!

    Beyond3D is proudly published by GPU Tools Ltd.
Loading...