9700 Pro < Ti4600 at MINIMUM fps - is this REALLY true?

Discussion in 'General 3D Technology' started by Joe Cool, Nov 9, 2002.

  1. Ailuros

    Ailuros Epsilon plus three
    Legend Subscriber

    Joined:
    Feb 7, 2002
    Messages:
    9,420
    Likes Received:
    179
    Location:
    Chania
    I don't exclude the possibility of a possible driver bug, but for fairness sake the texel fillrates between the cards are only 200MTps apart, and UT2003 is "just" a quad-textured game.
     
  2. gkar1

    Regular

    Joined:
    Jul 20, 2002
    Messages:
    614
    Likes Received:
    7
    This guy has been making the rounds thru rage3d and hardocp forums spreading flamebait in all of them. Even when confronted with hard facts refutting his argument, all he would post is that single link to Tom's I love Intel Hardware.
     
  3. KimB

    Legend

    Joined:
    May 28, 2002
    Messages:
    12,902
    Likes Received:
    218
    Location:
    Seattle, WA
    Bah, I've seen Tomshardware accused of bias on many sides of the fence, many times. I just think that he, and the others that write for that site, tend to get a bit emotional about the products they review, tend to jump to conclusions a bit too quickly, and tend to be very opinionated.

    Personally, I don't think they have any overriding bias. They're just abrasive and sometimes just plain don't pay attention.

    Just to get this off my chest, though, one of the most glaring issues that I've seen at Tomshardware, and a huge number of sites, is the continuing fallacy that ATI's anisotropic filtering is faster than nVidia's because it's adaptive.

    Every anisotropic filtering technique is adaptive.

    As a side note, it seems that the PR reps from both companies have been themselves misinformed (or something...) on this issue as well. I e-mailed Tomshardware (Van, actually, I think), and he said that both ATI and nVidia had actually confirmed this, but I really think that the people they talked to are just plain wrong. I forgot where I saw it, but I think there was a recent interview that finally seemed to put things together the way they really are. It basically stated that nVidia's technique doesn't always use a plain grid for anisotropic sampling, while ATI's does. This actually makes a lot of sense if you compare the two cards in a high-contrast region (the tech demo part of SS:SE is a perfect spot). Just when looking straight forward on this map, there is a decent amount of texture aliasing on the right and left portions of the screen on the Radeon 9700, but there isn't on the GeForce4. While this usually isn't noticeable in-game, I did notice it, in a big way, in one particular texture in Neverwinter Nights...that texture shimmered all over the place...
     
  4. JohnH

    Regular

    Joined:
    Mar 18, 2002
    Messages:
    586
    Likes Received:
    2
    Location:
    UK
    Not sure if this has already been mentioned, as I've only had time to read a couple of pages of the thread....

    Why not publish the median frame rate along with the average ? This tends to give a good indication of skews in the data set e.g a median that is much lower than the average indicates that typicale values are lower but the average is being raised by a smaller number of larger values, the reverse being true if the median is higher. So a "good" bm result would have a median that is higher than the average. Although I guess average and median being close gives you no additional information, in this case you need a histogram or perhaps just an std value in order to make any sense of the data.

    Regradinging other metrics, very interresting to many of the people who visit these forums, but are they really that useful to the average punter ? Does he really care that the median frame rate was lower because of polythroughput vs fillrate ?

    John.
     
  5. Nupraptor

    Newcomer

    Joined:
    Nov 11, 2002
    Messages:
    79
    Likes Received:
    0
    I'd agree with you. However, I still think it's a completely misinformative and lackluster site, and I cannot fathom why so many people swear by it.

    Given my already low opinion of Tom's Hardware, I take the entire issue with less than a grain of salt.
     
  6. Randell

    Randell Senior Daddy
    Veteran

    Joined:
    Feb 14, 2002
    Messages:
    1,869
    Likes Received:
    3
    Location:
    London
    Didnt Van leave Tom's ages ago?
     
  7. KimB

    Legend

    Joined:
    May 28, 2002
    Messages:
    12,902
    Likes Received:
    218
    Location:
    Seattle, WA
    Well, there are a hell of a lot of different ways to statistically-analyze the performance. Personally, I think a standard deviation approach would be the best way. That said, I'm really not sure precisely what form this should take in the math aspect, but here are a few ideas:

    1. A "frame score" should be something like (average framerate) - A*(standard deviation) where A is some number. What this will do is give a framerate above which a specific percentage of all frames are. For example, if A=1, then approximately 84% of the frames will be above the "frame score" in framerate. This should give a better indication of actual performance than minimum framerate scores.

    2. Reporting of "spike frequency" in framerate drops. Not entirely certain about the best way to do this, but it might be good to record a rather long demo in some game to use for a timedemo. This demo should be used in a very texture-intensive map, and a framerate graph should be calculated. The benchmark should record the number of severe drops in framerate (Say, the number of frames rendered at below 20 fps), at settings where the video card gets a rather high average framerate by comparison. This should be a good measurement of how much said video card stutters, primarily due to texture management issues.

    3. Alternate calculation of framerate/standard deviation data. Using either time per frame or framerate each frame for calculating averages and standard deviations may not be the best way to go. The simple fact is that the two numbers are related to one another by a 1/x function. It is easy to remove the non-linear nature of the relationship between these two by simply taking a logarithm of one or the other, then calculating mean and standard deviation data. This obviously requires per-frame information. The primary benefit of this sort of data collection is that it doesn't favor either high or low framerates. This would particularly good for comparing the standard deviation between different video cards, video cards which might not be performing on the same level, resulting in an excellent measurement of overall framerate stability.
     
  8. MrGaribaldi

    Regular

    Joined:
    Nov 23, 2002
    Messages:
    611
    Likes Received:
    0
    Location:
    In transit
    Basic> Thank you for responding :)

    Time is allways a problem when it comes to hobbies, and (I would guess) especially when building something just to run a test...

    But it was nice to see that someone else had come up with the idea of how to test FPS & such from the VGA out... Even more so, since you know how to do it :)

    And I'm sorry... I couldn't find any places that sell the attiny26 (nor any other attiny for that matter... )
     
  9. Magnum PI

    Veteran

    Joined:
    Feb 6, 2002
    Messages:
    1,054
    Likes Received:
    12
  10. Nagorak

    Regular

    Joined:
    Jun 20, 2002
    Messages:
    854
    Likes Received:
    0
    So you're saying they're a bad review site? I agree! 8)
     
  11. JohnH

    Regular

    Joined:
    Mar 18, 2002
    Messages:
    586
    Likes Received:
    2
    Location:
    UK
    Chalnoth, to be honest I think its just up to the reviewers to do a proper analysis of the data they gather and stop just publishing average rates. This probably just requires them to look at histograms of the frame rates and comment on any major deviations. Although I guess this may just get to technical for average Joe.
     
  12. LeStoffer

    Veteran

    Joined:
    Feb 6, 2002
    Messages:
    1,253
    Likes Received:
    13
    Location:
    Land of the 25% VAT
    If the reviewers were nice to us, they should just publish the darn data (e.g. via a link) so I could analyse the data myself. Everybody here should be able to reach a reasonable conclusion on their own. Note: hint for Dave Baumann. :wink:

    ... or it get's to difficult for average Joe-reviewer... It's much easier for them to stay out of trouble when they just report one average FPS figure. The other way demands comments that will probably awaken the fanboys.

    Not that I care - I'm all for it - but then I'm not the one who would have to answer some 1000+ flame mails. :wink:
     
  13. SLAGD

    Newcomer

    Joined:
    Dec 20, 2002
    Messages:
    3
    Likes Received:
    0
    Location:
    Canada
    New benchmark

    I have been developing a benchmark for ut2003 that uses the csv files for results. I have built in a function that will average the results per second instead of per 30th of a second like the normal ut2003 benchmarks. It graphs its own results including a line graph that give a second by second display of the framerate. I hope that this will provide a better way to benchmark ut2003. I am just finishing up some interface features and hope to release it before Christmas. If you have any questions or want to help beta test, you can email me at nicao@telus.net.
     
  14. ERP

    ERP Moderator
    Moderator Veteran

    Joined:
    Feb 11, 2002
    Messages:
    3,669
    Likes Received:
    49
    Location:
    Redmond, WA
    Re: New benchmark

    You might want to include Min/Max framerates/second aswell as the average in the graph. It would give some idea of the variance (coinsistency) of the framerate over the time interval.

    Just a suggestion.
     
  15. Joe DeFuria

    Legend

    Joined:
    Feb 6, 2002
    Messages:
    5,994
    Likes Received:
    70
    Wow...

    I almost forgot I was working on a similar tool (performance analysis based on the CSV file) I mentioned earlier in this thread...though I haven't really looked at it in about a month: Here's a beta of it:

    http://www.defuria.com/uttool/UTPA b0-5.zip

    I haven't had much time to document anything, but basically, here's the gist:

    I have NOT tried this with the latest UT 2003 patch, and I don't actually have UT 2003, so I cannot be certain if it still works with the latest patch. (If EPIC changed around the CSV file format, some things might not work properly.)


    * Requires MS Excel '97 or later. (Tested on '97 and '2000, should work on XP.)

    * After running a benchmark in UT, a CSV file is created that detailes the frame by frame performance. I have termed one CSV file as a "Data Set". Usually, you will change one factor between data sets, like resolution, AA / no AA, or even different video cards.

    * You import each data set into the tool. (Self explanatory using buttons.)

    * The tool has built-in routines for generating custom line and x-y graphs, as well as histograms. There is also a page that shows "quick stats" for each data set. (Max, min, and average frame rates, standard deviation, etc.)

    Have Fun!
     
  16. Joe DeFuria

    Legend

    Joined:
    Feb 6, 2002
    Messages:
    5,994
    Likes Received:
    70
    Here is some sample output. (Sorry for the large screen size, but they are gifs so should be pretty quick...)

    [​IMG]

    [​IMG]

    [​IMG]

    [​IMG]
     
  17. SLAGD

    Newcomer

    Joined:
    Dec 20, 2002
    Messages:
    3
    Likes Received:
    0
    Location:
    Canada
    ut2k3 benchmark

    Yes the tool does display minium and maximum results in the graphs as well as the in the text readout.
     
  18. MikeC

    Newcomer

    Joined:
    Feb 9, 2002
    Messages:
    194
    Likes Received:
    0
  19. dream caster

    Newcomer

    Joined:
    Jan 4, 2003
    Messages:
    40
    Likes Received:
    0
    Location:
    Chile
    I've followed this forums quite a time but never posted as i mostly come to learn (and as time passes by posts seem to be more understable to me) Now have an idea that one of the video experts from here could pick up, mostly about player perception of video speed.

    The FPS rate could be seen as another way to express the relevant variable: frame rendering time. There must be a relation between frame rendering time and and lag or stutter perception, so if only one frame takes longer and then game goes at higher speed nothing is perceived but if this goes on longer you'll see a jump or a stuttering; i.e. there is an umbral time to perceive video low speed wich is lower for lower speeds.
    It would be interesting to know exactly how it goes how does it affect each card or game.

    A way to roughly sum up would to display Times where X number of frames take longer than Y to render.
     
  20. Brent

    Regular

    Joined:
    Apr 11, 2002
    Messages:
    584
    Likes Received:
    4
    Location:
    Irving, TX
    i am getting more stable minimum fps now with dx9 and cat 3 drivers

    check this out, taken with our hardocp benchmarker in high quality

    it shows it in this order

    1600x1200
    1280x1024
    1024x768

    for each map going down

    dm-antalus
    1.437771 / 48.869598 / 333.726654 fps
    Score = 48.878502

    dm-antalus
    2.067634 / 72.455894 / 215.843491 fps
    Score = 72.470192

    dm-antalus
    3.284277 / 113.696815 / 334.620483 fps
    Score = 113.712746

    dom-suntemple
    32.823185 / 58.730614 / 185.247437 fps
    Score = 58.755745

    dom-suntemple
    19.700747 / 84.117439 / 244.856949 fps
    Score = 84.153282

    dom-suntemple
    41.442604 / 125.888641 / 377.307068 fps
    Score = 125.936600

    dm-phobos2
    12.671028 / 62.678513 / 392.257690 fps
    Score = 62.724178

    dm-phobos2
    18.701948 / 85.040565 / 518.236938 fps
    Score = 85.102310

    dm-phobos2
    28.843964 / 119.140808 / 497.114471 fps
    Score = 119.219048

    dm-inferno
    25.464371 / 43.315151 / 118.274788 fps
    Score = 43.362762

    dm-inferno
    15.921018 / 60.744934 / 566.414368 fps
    Score = 60.813572

    dm-inferno
    20.075624 / 92.403603 / 260.063019 fps
    Score = 92.510666

    ctf-face3
    37.914314 / 81.021378 / 414.508575 fps
    Score = 81.102463

    ctf-face3
    37.567230 / 106.426689 / 466.579559 fps
    Score = 106.516747

    ctf-face3
    57.523342 / 138.403870 / 489.575836 fps
    Score = 138.481323

    ctf-citadel
    42.989880 / 67.262878 / 190.795486 fps
    Score = 67.327293

    ctf-citadel
    28.994799 / 91.049538 / 253.042252 fps
    Score = 91.200172

    ctf-citadel
    46.869839 / 116.777359 / 305.874390 fps
    Score = 117.001839

    dm-asbestos
    41.360710 / 77.156357 / 284.155609 fps
    Score = 77.207275

    dm-asbestos
    57.018593 / 107.473923 / 352.990845 fps
    Score = 107.522034

    dm-asbestos
    81.198792 / 160.974304 / 479.018585 fps
    Score = 161.114685


    Only Antalus is bottoming out real low, but the others are where they should be!

    I also ran http://www.telusplanet.net/public/bentiah/utbench/utbench2k3_1.htm and the results taken with this benchmark also show Antalus bottoming out but the others clocking in fine

    I also looked and his results he is pulling from the CSV's are almost identical to the results in the lowframerate.log file, in fact they are only 1 fps different at most, and usually even less then an fps different then what is shown in the lowframerate.log file

    so to me that suggests using the results from that txt file are fine since they match what the csv's tell us for min/avg/max


    btw testing machine was

    abit kd7-raid
    axp 2400+
    radeon 9500 pro
    winxp
    dx9, cat 3.0
     
Loading...

Share This Page

  • About Us

    Beyond3D has been around for over a decade and prides itself on being the best place on the web for in-depth, technically-driven discussion and analysis of 3D graphics hardware. If you love pixels and transistors, you've come to the right place!

    Beyond3D is proudly published by GPU Tools Ltd.
Loading...