9700 Pro < Ti4600 at MINIMUM fps - is this REALLY true?

I don't exclude the possibility of a possible driver bug, but for fairness sake the texel fillrates between the cards are only 200MTps apart, and UT2003 is "just" a quad-textured game.

 

This guy has been making the rounds thru rage3d and hardocp forums spreading flamebait in all of them. Even when confronted with hard facts refutting his argument, all he would post is that single link to Tom's I love Intel Hardware.

 

Bah, I've seen Tomshardware accused of bias on many sides of the fence, many times. I just think that he, and the others that write for that site, tend to get a bit emotional about the products they review, tend to jump to conclusions a bit too quickly, and tend to be very opinionated.

Personally, I don't think they have any overriding bias. They're just abrasive and sometimes just plain don't pay attention.

Just to get this off my chest, though, one of the most glaring issues that I've seen at Tomshardware, and a huge number of sites, is the continuing fallacy that ATI's anisotropic filtering is faster than nVidia's because it's adaptive.

Every anisotropic filtering technique is adaptive.

As a side note, it seems that the PR reps from both companies have been themselves misinformed (or something...) on this issue as well. I e-mailed Tomshardware (Van, actually, I think), and he said that both ATI and nVidia had actually confirmed this, but I really think that the people they talked to are just plain wrong. I forgot where I saw it, but I think there was a recent interview that finally seemed to put things together the way they really are. It basically stated that nVidia's technique doesn't always use a plain grid for anisotropic sampling, while ATI's does. This actually makes a lot of sense if you compare the two cards in a high-contrast region (the tech demo part of SS:SE is a perfect spot). Just when looking straight forward on this map, there is a decent amount of texture aliasing on the right and left portions of the screen on the Radeon 9700, but there isn't on the GeForce4. While this usually isn't noticeable in-game, I did notice it, in a big way, in one particular texture in Neverwinter Nights...that texture shimmered all over the place...

 

Not sure if this has already been mentioned, as I've only had time to read a couple of pages of the thread....

Why not publish the median frame rate along with the average ? This tends to give a good indication of skews in the data set e.g a median that is much lower than the average indicates that typicale values are lower but the average is being raised by a smaller number of larger values, the reverse being true if the median is higher. So a "good" bm result would have a median that is higher than the average. Although I guess average and median being close gives you no additional information, in this case you need a histogram or perhaps just an std value in order to make any sense of the data.

Regradinging other metrics, very interresting to many of the people who visit these forums, but are they really that useful to the average punter ? Does he really care that the median frame rate was lower because of polythroughput vs fillrate ?

John.

 

I'd agree with you. However, I still think it's a completely misinformative and lackluster site, and I cannot fathom why so many people swear by it.

Given my already low opinion of Tom's Hardware, I take the entire issue with less than a grain of salt.

 

Didnt Van leave Tom's ages ago?

 

Well, there are a hell of a lot of different ways to statistically-analyze the performance. Personally, I think a standard deviation approach would be the best way. That said, I'm really not sure precisely what form this should take in the math aspect, but here are a few ideas:

1. A "frame score" should be something like (average framerate) - A*(standard deviation) where A is some number. What this will do is give a framerate above which a specific percentage of all frames are. For example, if A=1, then approximately 84% of the frames will be above the "frame score" in framerate. This should give a better indication of actual performance than minimum framerate scores.

2. Reporting of "spike frequency" in framerate drops. Not entirely certain about the best way to do this, but it might be good to record a rather long demo in some game to use for a timedemo. This demo should be used in a very texture-intensive map, and a framerate graph should be calculated. The benchmark should record the number of severe drops in framerate (Say, the number of frames rendered at below 20 fps), at settings where the video card gets a rather high average framerate by comparison. This should be a good measurement of how much said video card stutters, primarily due to texture management issues.

3. Alternate calculation of framerate/standard deviation data. Using either time per frame or framerate each frame for calculating averages and standard deviations may not be the best way to go. The simple fact is that the two numbers are related to one another by a 1/x function. It is easy to remove the non-linear nature of the relationship between these two by simply taking a logarithm of one or the other, then calculating mean and standard deviation data. This obviously requires per-frame information. The primary benefit of this sort of data collection is that it doesn't favor either high or low framerates. This would particularly good for comparing the standard deviation between different video cards, video cards which might not be performing on the same level, resulting in an excellent measurement of overall framerate stability.

 

Basic> Thank you for responding

Time is allways a problem when it comes to hobbies, and (I would guess) especially when building something just to run a test...

But it was nice to see that someone else had come up with the idea of how to test FPS & such from the VGA out... Even more so, since you know how to do it

And I'm sorry... I couldn't find any places that sell the attiny26 (nor any other attiny for that matter... )

 

did you see the latest "Special THG supplement" about pentium 4 HT ?

http://www.tomshardware.com/cpu/20021202/index.html

it should have been on intel's site, not there..

 

So you're saying they're a bad review site? I agree! 8)

 

Chalnoth, to be honest I think its just up to the reviewers to do a proper analysis of the data they gather and stop just publishing average rates. This probably just requires them to look at histograms of the frame rates and comment on any major deviations. Although I guess this may just get to technical for average Joe.

 

If the reviewers were nice to us, they should just publish the darn data (e.g. via a link) so I could analyse the data myself. Everybody here should be able to reach a reasonable conclusion on their own. Note: hint for Dave Baumann.

... or it get's to difficult for average Joe-reviewer... It's much easier for them to stay out of trouble when they just report one average FPS figure. The other way demands comments that will probably awaken the fanboys.

Not that I care - I'm all for it - but then I'm not the one who would have to answer some 1000+ flame mails.

 

New benchmark

I have been developing a benchmark for ut2003 that uses the csv files for results. I have built in a function that will average the results per second instead of per 30th of a second like the normal ut2003 benchmarks. It graphs its own results including a line graph that give a second by second display of the framerate. I hope that this will provide a better way to benchmark ut2003. I am just finishing up some interface features and hope to release it before Christmas. If you have any questions or want to help beta test, you can email me at nicao@telus.net.

 

Re: New benchmark

You might want to include Min/Max framerates/second aswell as the average in the graph. It would give some idea of the variance (coinsistency) of the framerate over the time interval.

Just a suggestion.

 

Wow...

I almost forgot I was working on a similar tool (performance analysis based on the CSV file) I mentioned earlier in this thread...though I haven't really looked at it in about a month: Here's a beta of it:

http://www.defuria.com/uttool/UTPA b0-5.zip

I haven't had much time to document anything, but basically, here's the gist:

I have NOT tried this with the latest UT 2003 patch, and I don't actually have UT 2003, so I cannot be certain if it still works with the latest patch. (If EPIC changed around the CSV file format, some things might not work properly.)


* Requires MS Excel '97 or later. (Tested on '97 and '2000, should work on XP.)

* After running a benchmark in UT, a CSV file is created that detailes the frame by frame performance. I have termed one CSV file as a "Data Set". Usually, you will change one factor between data sets, like resolution, AA / no AA, or even different video cards.

* You import each data set into the tool. (Self explanatory using buttons.)

* The tool has built-in routines for generating custom line and x-y graphs, as well as histograms. There is also a page that shows "quick stats" for each data set. (Max, min, and average frame rates, standard deviation, etc.)

Have Fun!

 

Here is some sample output. (Sorry for the large screen size, but they are gifs so should be pretty quick...)

 

ut2k3 benchmark

Yes the tool does display minium and maximum results in the graphs as well as the in the text readout.

 

UT2K3Bench 1.31

 

I've followed this forums quite a time but never posted as i mostly come to learn (and as time passes by posts seem to be more understable to me) Now have an idea that one of the video experts from here could pick up, mostly about player perception of video speed.

The FPS rate could be seen as another way to express the relevant variable: frame rendering time. There must be a relation between frame rendering time and and lag or stutter perception, so if only one frame takes longer and then game goes at higher speed nothing is perceived but if this goes on longer you'll see a jump or a stuttering; i.e. there is an umbral time to perceive video low speed wich is lower for lower speeds.
It would be interesting to know exactly how it goes how does it affect each card or game.

A way to roughly sum up would to display Times where X number of frames take longer than Y to render.

 

i am getting more stable minimum fps now with dx9 and cat 3 drivers

check this out, taken with our hardocp benchmarker in high quality

it shows it in this order

1600x1200
1280x1024
1024x768

for each map going down

dm-antalus
1.437771 / 48.869598 / 333.726654 fps
Score = 48.878502

dm-antalus
2.067634 / 72.455894 / 215.843491 fps
Score = 72.470192

dm-antalus
3.284277 / 113.696815 / 334.620483 fps
Score = 113.712746

dom-suntemple
32.823185 / 58.730614 / 185.247437 fps
Score = 58.755745

dom-suntemple
19.700747 / 84.117439 / 244.856949 fps
Score = 84.153282

dom-suntemple
41.442604 / 125.888641 / 377.307068 fps
Score = 125.936600

dm-phobos2
12.671028 / 62.678513 / 392.257690 fps
Score = 62.724178

dm-phobos2
18.701948 / 85.040565 / 518.236938 fps
Score = 85.102310

dm-phobos2
28.843964 / 119.140808 / 497.114471 fps
Score = 119.219048

dm-inferno
25.464371 / 43.315151 / 118.274788 fps
Score = 43.362762

dm-inferno
15.921018 / 60.744934 / 566.414368 fps
Score = 60.813572

dm-inferno
20.075624 / 92.403603 / 260.063019 fps
Score = 92.510666

ctf-face3
37.914314 / 81.021378 / 414.508575 fps
Score = 81.102463

ctf-face3
37.567230 / 106.426689 / 466.579559 fps
Score = 106.516747

ctf-face3
57.523342 / 138.403870 / 489.575836 fps
Score = 138.481323

ctf-citadel
42.989880 / 67.262878 / 190.795486 fps
Score = 67.327293

ctf-citadel
28.994799 / 91.049538 / 253.042252 fps
Score = 91.200172

ctf-citadel
46.869839 / 116.777359 / 305.874390 fps
Score = 117.001839

dm-asbestos
41.360710 / 77.156357 / 284.155609 fps
Score = 77.207275

dm-asbestos
57.018593 / 107.473923 / 352.990845 fps
Score = 107.522034

dm-asbestos
81.198792 / 160.974304 / 479.018585 fps
Score = 161.114685


Only Antalus is bottoming out real low, but the others are where they should be!

I also ran http://www.telusplanet.net/public/bentiah/utbench/utbench2k3_1.htm and the results taken with this benchmark also show Antalus bottoming out but the others clocking in fine

I also looked and his results he is pulling from the CSV's are almost identical to the results in the lowframerate.log file, in fact they are only 1 fps different at most, and usually even less then an fps different then what is shown in the lowframerate.log file

so to me that suggests using the results from that txt file are fine since they match what the csv's tell us for min/avg/max


btw testing machine was

abit kd7-raid
axp 2400+
radeon 9500 pro
winxp
dx9, cat 3.0