what benchmark results matter to you?

[PP]

When I talk about percentages I mean every result is matched against highest result, and you get a relative result. For example, if you search for x800pe, x800pro, 9800xt, 9800pro and 9600xt in tr:aod

http://www.usuarios.lycos.es/gpumania/rendimiento.php?gd=1&ib=1274

9600xt is 28% of x800 pe frame rate.

That can be misleading though without knowing how fast the x800pe is going. If most other benchmarks scored 50fps for the x800pe while you got 200fps (perhaps through error), the implication would be that you also got 50fps (since you don't say) which then implies that the 28% 9600xt got ~15 fps, while in actuality it might have gotten more like 60fps. By hiding the actual scores, your reader is more likely to be mislead by errors than if the scores are shown.

Nite_Hawk

I know this kind of representation could be misleading, but the reason behind hiding scores is only ethics. That arent my benchmarks, and if people want to know fps only need to click the link.

 

[Nite_Hawk]

ahh, okay. Actually, I'm planning on just having a few systems that fit into "ancient" "low-end" "mid-range" "high-end" and "workstation" categories. And I'm going to be testing just "graphics cards" like you would buy out of the box. I plan on trying to get every card (not brand, of course) into the database. It's a mighty-big task, but I think it will be well-worth it.

of course, resolution and AA/AF would also be considered "dimensions" yes? Also, there is the issue of driver versions... well, I have given it some thought and I think I'll only have one driver version posted. I'll do tests and keep records of other driver versions, but only use the ones that give the best scores and pass my quality requirements (no cheats! ) and then show which ones you're seeing the results for.

oh, and no I haven't taken probability. I haven't even graduated high school much less taken any college.... and probably never will graduate

That'd probably work pretty well just segmenting the different systems into different age catagories. As far as demensionality goes, it's really up to you how detailed you want it to be. A really simple set of dimensions could be:

1) Computer Type (Ancient, Low-end, mid-range, High-end, workstation)
2) Video Card (9700pro, ti4600, etc)
3) Resolution (etc)
4) Quality Settings (High, medium, Llow)
5) Benchmark (3DMark03, Quake3, etc)

So in this case you might be able to get by with 5 dimensions. This doesn't tell you a lot about how a card plays with AA and with Anisotropic filtering though, so you could split up Quality Settings into AA, Texture Filtering, and any other attributes you may want (perhaps texture compression or something). You could also break up videocard into the GPU, the GPU speed, the video memory speed, etc.

I decided to go for a more intimate look at the data and ended up with 14 attributes:

CPU Type (PIII, PIV, Athlon, etc)
CPU Speed (in Mhz)
RAM Speed (in MB/s)
Chipset (nForce2, i850E, etc)
GPU Type (NV30, R300, etc)
GPU Speed (in MHz)
VRAM Speed (in MB/s)
Driver (Cat4.6, etc)
Resolution (1024x768, etc)
AA (2x, 4x, etc)
Aniso (2x, 4x, etc)
Filtering (Bi, bri, tri, etc)
Map (Antalus, Cathedral, etc)
Benchmark (Quake3, SS:SE, etc)

Note that a number of these could probably be combined. There isn't much reason for having map and benchmark seperate (except that in some cases I have the same game listed under two benchmarks, UT2003_Botmatch and UT2003_Flyby for example).

With this many attributes, there are a *lot* of possible combinations of hardware and software to test.

Btw, it's not a big deal that you arn't out of highschool. I did a lot of raytracing back then, you have more free time now than you'll probably have again for a long time, so make good use of it! If you are interested in picking up some basic statistics, find a copy of "The Basic Practice of Statistics" by David S. Moore. It's non-calculus based, so if you haven't had calc yet it's not too big of a deal. You'll want to understand the binomial theorem, and the concepts behind statistical significance and how you find it. This way you can let people know if the results you get are meaningful.

Nite_Hawk

 

[Nite_Hawk]

I know this kind of representation could be misleading, but the reason behind hiding scores is only ethics. That arent my benchmarks, and if people want to know fps only need to click the link.

I guess it seems like it might be more of a disservice to the author by not including their exact scores, because you might end up misrepresenting their results by only providing the percentages rather than the actual numbers they got. Why do you feel that posting the actual scores would be unethical?

Nite_Hawk

 

[Sage]

Btw, it's not a big deal that you arn't out of highschool. I did a lot of raytracing back then, you have more free time now than you'll probably have again for a long time, so make good use of it! If you are interested in picking up some basic statistics, find a copy of "The Basic Practice of Statistics" by David S. Moore. It's non-calculus based, so if you haven't had calc yet it's not too big of a deal. You'll want to understand the binomial theorem, and the concepts behind statistical significance and how you find it. This way you can let people know if the results you get are meaningful.

nono, you misunderstand. I am out of high school, have been for two years. I'm going to be 20 in Aug. I just didn't finish high school. And, as much as I'd love to pick up that book, I would just end up like every other book I've ever bought- in the pile. I can't read very well, there's no way I could possibly get through that book. I can read comics although I find them very borish. Other than that, I just can't do it. :?

 

[Nite_Hawk]

nono, you misunderstand. I am out of high school, have been for two years. I'm going to be 20 in Aug. I just didn't finish high school. And, as much as I'd love to pick up that book, I would just end up like every other book I've ever bought- in the pile. I can't read very well, there's no way I could possibly get through that book. I can read comics although I find them very borish. Other than that, I just can't do it. :?

Having degrees and everything is rather over-rated anyway. In a lot of ways I wish they'd do away with them entirely as they don't really judge how smart you are, or even how well you know what you are doing.

Anyway, if people really want to know the statistical significance stuff (probably a pretty small percentage of the people reading your site) they can figure it out themselves if you give them the numbers. It's neat stuff to know because it lets you say with certainty that one card is faster than the other, but statistics isn't for everyone (I hate most normal math, but love statistics so I'm kind of wierd).

The stuff that probably would be the most useful is to:

A) Make sure you run the tests a couple of times and average the results. Providing a standard deviation would be really useful too, but if you provide all of the results and the mean, they can figure it out themselves too.

B) Say *exactly* what settings you used for each test. So far Dave, and Brent over at HardOCP seem to be the only ones that consistently are doing this.

C) Make sure to indicate any times you are reusing old test results so that if people are using them they know not to count it twice.

Anyway, good luck on the project! Glad to see someone else is interested in this kind of stuff!

Nite_Hawk

 

[overclocked]

I voted on the last when you can change resoultion and AA/AF, I think this would be a good idea atleast for me.