what benchmark results matter to you?
- Thread starter Sage
- Start date
Hi Sage,
I'm currently working on something rather similar, though a bit different. I collected about 3600 different "scores" from 40 different reviews off the internet. I limited the benchmarks to 3DMark01, 3DMark03, Quake3, SS:SE, RTCW, UT2003 (Botmatch) and UT2003 (Flyby). Most of the videocards were ATI 9700/9600 series, though there were also GF4 and NV30/NV35 cards in there as well. I'm currently performing some data mining work on them. I wrote a paper last spring about it which you can see the slides for here:
http://www.subpixel.org/~nh/subpixel/Slides/
or a zip archive:
http://www.subpixel.org/~nh/Mark_Nelson_Slides_Final.zip
And you can see the paper here:
http://www.subpixel.org/~nh/subpixel/subpixel_paper.doc
Currently I'm working on extending my research. Just last night I ran through 637 different classification tests using all combiniations of 2 attributes from 13 total across 7 different benchmarks. The purpose of this is to find out in the dataset how related different attributes are to each other, so that I can then try to find out exactly how correlated attributes are to predicting the score. (IE, how important is knowing the CPU speed of a system when determining the framerate in Quake3?)
Eventually the goal of this project is to maintain a searchable online database of scores (framerates or 3DMark scores) that have references back to the original articles they were pulled from, but also to use those numbers to try and create predictive models to guess about how various configurations "should" score. You could use this to figure out if the scores you are getting are reasonable given your hardware/software configuration.
Nite_Hawk
I'm currently working on something rather similar, though a bit different. I collected about 3600 different "scores" from 40 different reviews off the internet. I limited the benchmarks to 3DMark01, 3DMark03, Quake3, SS:SE, RTCW, UT2003 (Botmatch) and UT2003 (Flyby). Most of the videocards were ATI 9700/9600 series, though there were also GF4 and NV30/NV35 cards in there as well. I'm currently performing some data mining work on them. I wrote a paper last spring about it which you can see the slides for here:
http://www.subpixel.org/~nh/subpixel/Slides/
or a zip archive:
http://www.subpixel.org/~nh/Mark_Nelson_Slides_Final.zip
And you can see the paper here:
http://www.subpixel.org/~nh/subpixel/subpixel_paper.doc
Currently I'm working on extending my research. Just last night I ran through 637 different classification tests using all combiniations of 2 attributes from 13 total across 7 different benchmarks. The purpose of this is to find out in the dataset how related different attributes are to each other, so that I can then try to find out exactly how correlated attributes are to predicting the score. (IE, how important is knowing the CPU speed of a system when determining the framerate in Quake3?)
Eventually the goal of this project is to maintain a searchable online database of scores (framerates or 3DMark scores) that have references back to the original articles they were pulled from, but also to use those numbers to try and create predictive models to guess about how various configurations "should" score. You could use this to figure out if the scores you are getting are reasonable given your hardware/software configuration.
Nite_Hawk
gsgrunt said:
Yes, it is actually legal as you can't copyright facts (This benchmark scored this amount with this configuration). Even at that, it'd be covered under fair use to verify those results assuming you give credit to the author and don't plagerize. This kind of thing has been covered in the academic field for years.
Nite_Hawk
that's a very interesting idea. the purpose of my site was more so that people could do an easy comparrison of video cards they are thinking of buying, or even including their current one. So many people ask things like "how much faster in UT2K4 will the 6800U be over my 9500 Pro" but you'll be hard-pressed to even find benchmarks of those two cards using the same setup, much less in the same article.
Sage said:Nite_Hawk said:
hey, now that lifts a great burden from my shoulders! I was afraid that I was gonna have to buy a new system and a bunch of viddy cards to get the thing started
Just keep in mind I'm not a lawyer or anything, so don't take that as legal advice.
Afaik though, you should be in the clear so long as you don't present the data as your own and provide credit back to the original author.Nite_Hawk
When I made my site I was thinking in the copyright issue. Taking data from other sites maybe is´nt illegal, but I should be done with some ethics. I did in a way where no fps is shown, only %, and of course, link to source is a must (ethics, and maybe people want to know if data is right ).
).PP said:When I made my site I was thinking in the copyright issue. Taking data from other sites maybe is´nt illegal, but I should be done with some ethics. I did in a way where no fps is shown, only %, and of course, link to source is a must (ethics, and maybe people want to know if data is right
yeah, first of all you have to make sure that their testing procedures are acceptable. many sites are lazy and dont practice safe se...err... testing. also, I'd just feel bad about using a sites results without their permission. those two really narrow down your possibilities.
PP said:When I made my site I was thinking in the copyright issue. Taking data from other sites maybe is´nt illegal, but I should be done with some ethics. I did in a way where no fps is shown, only %, and of course, link to source is a must (ethics, and maybe people want to know if data is right
I think so long as the author is credited for their findings it is perfectly reasonable to cite the exact number, especially so that there is no misrepresentation of what it is exactly that they found. Playing the percentage game is dangerous because it can make their results look different than they actually are (unless you provide your numbers, in which case it's trivial to find out what their numbers were anyway).
Specifically in this case, it sounds like Sage wants to create a "google" of sorts for videocards, where it is easy to find reviews that are testing certain videocards with specific configurations or benchmarks.
Nite_Hawk
Sage said:
It sounds like a good project. The hardest parts will be collecting the data and insuring statistical significance for your results. Similarly, you'll end up with holes where old cards won't often be tested with new CPUs, and new cpus won't be tested with old cards (similar situations exist with drivers, ram, and even the games being tested). This is actually a large part of the reason why I'm trying to build the predictive model. I want to be able to guess how good a given card does given how well other cards do even if there is no data for that specific benchmark (but we can perhaps extrapolate it from how well it, and other cards do in other benchmarks, and how correlated different attributes are with that benchmark). The end goal is to be able to let people choose system configurations and it will tell them within some region of error how it will perform on a given benchmark.
Nite_Hawk
Nite_Hawk said:
actually, I plan on building a few systems and doing the benchmarks myself once I get the money together. I don't really have living expenses so any revenue generated from the site goes to paying the costs of running it, and then into systems and cards to do the benchmarking on. And I do plan on keeping the old cards around and retesting them with newer games and faster CPU's, as well as testing the newer cards on older games and CPU's. So, actually kind of the same thing you are doing but through a different means
When I talk about percentages I mean every result is matched against highest result, and you get a relative result. For example, if you search for x800pe, x800pro, 9800xt, 9800pro and 9600xt in tr:aod
http://www.usuarios.lycos.es/gpumania/rendimiento.php?gd=1&ib=1274
9600xt is 28% of x800 pe frame rate.
http://www.usuarios.lycos.es/gpumania/rendimiento.php?gd=1&ib=1274
9600xt is 28% of x800 pe frame rate.
Sage said:actually, I plan on building a few systems and doing the benchmarks myself once I get the money together. I don't really have living expenses so any revenue generated from the site goes to paying the costs of running it, and then into systems and cards to do the benchmarking on. And I do plan on keeping the old cards around and retesting them with newer games and faster CPU's, as well as testing the newer cards on older games and CPU's. So, actually kind of the same thing you are doing but through a different means
That's a much better way to do it if you can afford it and can take the time. Granted you'll need a lot of systems to test with, but you'll be able to make sure you don't have holes in the data. How many dimensions do you plan to test across? I settled on about 14, but I'm sure that could be reduced. Actually, now that I've finished the tests I just ran last night, I should be able to tell you of those 14 how many have a strong co-dependence and may be possible to elliminate.
Make sure to setup your relational schema well before writing any of the code. I had to redo mine a couple of times. I ended up settling on a star schema with score at the center and most of the other attributes surrounding the center. I haven't worked on that in a while though. Right now I'm using flat files as weka (the data mining tool I'm using) uses those.
Nite_Hawk
Nite_Hawk said:
I don't know about all of this stuff. I'm actually very ignorant when it comes to database anything. I'm really good at comming up with ideas but, I have to have other people who actually know what they are doing to make stuff happen.
Remember, I'm not doing any of the coding becuase... well... because I'm a fsking idiot and no matter how I try I simply can't learn any kind of coding. Am I in over my head? Meh, maybe I am... I usually do that.
PP said:
That can be misleading though without knowing how fast the x800pe is going. If most other benchmarks scored 50fps for the x800pe while you got 200fps (perhaps through error), the implication would be that you also got 50fps (since you don't say) which then implies that the 28% 9600xt got ~15 fps, while in actuality it might have gotten more like 60fps. By hiding the actual scores, your reader is more likely to be mislead by errors than if the scores are shown.
Nite_Hawk
Sage said:
Heh, that's ok. The dimensionality stuff just means how many different things you will change. For example, if you tested with all the same settings except that the CPU speed changed between tests, you would have 1 dimensional data. If both the CPU speed and the CPU type changed, it might be 2 dimensional data. You can also combine them so that you have "CPU Type and CPU speed" as a single dimension, but there are downsides sometimes to doing that. By changing the number of dimensions, you can decide how intimately you want to look at the data. You could for example, make "video card" a dimension, but you could just as easily give "Video memory speed" "GPU Speed" "GPU Type" "AGP Speed" and "Number of Pipelines" each their own dimension.
Have you taken any probabilty yet? It's extremely useful for this kind of stuff. Database knowledge is useful, but not nearly as useful as statistics. Really, you could do this all with flat files (or an excel spreadsheet) if you you really wanted to. The more important thing is setting up how you are going to run the tests, and examining how accuruate your results are, and whether or not they are meaningful.
Nite_Hawk
Nite_Hawk said:
ahh, okay. Actually, I'm planning on just having a few systems that fit into "ancient" "low-end" "mid-range" "high-end" and "workstation" categories. And I'm going to be testing just "graphics cards" like you would buy out of the box. I plan on trying to get every card (not brand, of course) into the database. It's a mighty-big task, but I think it will be well-worth it.
of course, resolution and AA/AF would also be considered "dimensions" yes? Also, there is the issue of driver versions... well, I have given it some thought and I think I'll only have one driver version posted. I'll do tests and keep records of other driver versions, but only use the ones that give the best scores and pass my quality requirements (no cheats!
) and then show which ones you're seeing the results for.oh, and no I haven't taken probability. I haven't even graduated high school
much less taken any college.... and probably never will graduate Similar threads
