Just a bit of a tease....

[Nite_Hawk]

Hi Guys,

So I'm working on a project that I hope a lot of people will be able to find useful. I'm not going to say too much about it, except that I hope we'll be able to find a lot of useful information out with it. Anyway, I thought I'd give you guys a tease to see how people's reactions are. This isn't by any means complete, and charts like these will be created in realtime for the viewer, not static like this picture. Can you guess what card, benchmark, and settings are being shown in the chart below?

Nite_Hawk

 

[K.I.L.E.R]

Have you got any meaningless maths to backup your claim/s?

j/k man.

Are you benchmarking 3d hardware sites?
Beyond3D should be getting over 1Tfps.

 

[Grall]

Can you guess what card, benchmark, and settings are being shown in the chart below?

That question is kind of in the same vein as, "can you breathe?"; of course we can GUESS! A better question would be, "can you guess correctly (on your first attempt), what...". Answer would be most probably no, not from that little information.

Anyway, what's the point of all this?


*G*

 

[K.I.L.E.R]

For all we know he could be benchmarking a Voodoo 3 rendering 1 triangle.

 

[akira888]

card: 3dfx Rampage in quad-chip configuration
test: Microsoft DX5 DirectDraw Compatibility Test

 

[Nite_Hawk]

Bah, you people are no fun! I was typing that after like 8 hours of coding and collecting benchmark scores. It's only logical that I wouldn't make much sense.

K.I.L.I.R has the right idea though. I'll give you all a hint. It's a popular card, a popular benchmark, and a popular setting. Right now there are only about 40 sites results entered in, but I think this could easily turn into a hundred or two with some proper research. Already you can see things lumping in the 155-165 fps range, and around 180fps. From what I remember of the data, it seems to correlate to CPU architecture (athlon versus P4). Notice the outliers, one exceptionally slow, and two exceptionally fast. Personally I think that's rather interesting. It will be more interesting once driver revisions and cpu architecture/speed are accounted for, and we have many more samples. Especially when this is done for many benchmarks, quality settings, and videocards.

(BTW, if this still isn't clear, the chart is showing a histogram of FPS values for review sites. When a review site's benchmark score falls in a certain range, it raises the corresponding bar for that range by 1. This way, you can easily see what ranges of scores contain the most number of scores. This is important because with enough samples, you can tell aproximately how fast the card performed for the majority of review sites, and what the deviation is from the mean.)

Nite_Hawk

 

[Bouncing Zabaglione Bros.]

This would be really good if it came with a list of site names, and when you click on a sit, it lights up that site's "score bar". Even better if you could wave your mouse over the bars and have a list of sites come up, so you could get instant feedback on who scored what at any given point.

 

[Nite_Hawk]

Bouncing:

That's the plan basically. Each review score will have a bibliography entry, with information as to the publishing date, the score, the cpu used, the drivers, and it will automatically append any notes about drivers being encrypted, any "optimizations" used, any bugs, and I'll have an additional section discriminating anti-detector.

The goal is you could look at all scores for a specific GPU to see how clock speed changes and/or drivers change the score. You could look at scores for all GPUs running on a PIV to see the range of scores. We can use the stddev and stderr to see what kind of score should be expected for a given system.

I'm *really* excited about this.

Nite_Hawk

 

[Reverend]

I am totally and utterly frustrated with writing a readme for a new game+demo benchmark, and absolutely frustrated with trying to record a demo in one of the game's levels so I am therefore a little dense right now and do not really know what this is about, but I do know it belongs in another forum.

 

[Nite_Hawk]

Rev: I guess this forum makes some sense, though I wasn't sure if this was better or 3D tech was better. It's talking about technology but in something of an industry context.

Anyway, here's the basic idea. You have a lot, probably over a hundred, sites doing videocard reviews. Often times, a single site will have on record somewhere around 5 different scores for a given benchmark for the same vidoecard over multiple reviews (a lot of sites re-test a card when they do shootout type reviews with other cards). With all of this information, you can start looking at trends. You can see how scores change across cards, drivers, cpus, and review sites. You can even do things like compare how much of a standard deviation there is when looking at one brand of cards versus another brand of cards. Why is there that much deviation? Is it consistent across different cpu architectures? Different benchmarks?

There is *a lot* of information out there that is not being used well right now. I want to start using it and see if we do hypothesis testing and come to any kind of quantifiable conclusions.

Nite_Hawk

 

[Jerky]

Here's my guess:

3DMark 2001SE: Car Chase or Lobby on a FX5900.

How do you account for differences in driver settings (i.e. performance, quality, mipmap levels etc.)?

- Jonathan.

 

[Nite_Hawk]

Jerky:

Good guess, but not correct. I'm not entirely sure how I'm going to deal with texture quality yet. Should it be a search option? Should I simply list what quality setting they used when comparing scores?

So far one of the biggest problems I've had is that I have to throw away reviews that don't specifically state what settings they are using. One review didn't even say which map was being used for the test! (if that gives you any hint)

Nite_Hawk

 

[Jerky]

Jerky:

Good guess, but not correct. I'm not entirely sure how I'm going to deal with texture quality yet. Should it be a search option? Should I simply list what quality setting they used when comparing scores?

So far one of the biggest problems I've had is that I have to throw away reviews that don't specifically state what settings they are using. One review didn't even say which map was being used for the test! (if that gives you any hint)

Nite_Hawk

Ok. Here's my second guess:

UT2K3 - DM-Antalus flyby. Max Details/Ultra Details on FX5900.

I think texture quality is an important thing that should be searchable. In changing between performance to quality, you can get a huge difference in benchmark scores (which may explain the two poles in your chart above).

If it is Anatalus flyby, then all my reviews I've written won't be included. I use CTF-Face3 and DM-Asbestos. I never thought that DM-Antalus was a good reflection of the maps people play on UT2K3 or where maps were heading (i.e. larger outdoor environments).

Is there any way to search by map as well?

 

[Nite_Hawk]

Jerky: Very close! Try the other side of the fence and take a step back!

Your right about the map though. It's Anatalus, and I mostly just picked it because it was a common benchmark. Your right, your review scores arn't there (yet). I'm planning on having different maps be considered different benchmarks for this project, so I'd probably list 3 UT2K3 maps under the UT2K3 heading, the same with Q3, and other benchmarks as well. I'll probably use the most common ones just to get the sampling up, so that means 3DMark03, UT2003, Q3, Sereous Sam, and maybe JKII or Wolf3D. Cammanche 4 seems pretty popular too. Any other suggestions?

Really, using mysql, it shouldn't be too tough to make pretty much everything searchable...

Nite_Hawk

 

[Jerky]

This is beginning to feel like scooping fish with a net. Looks easy, but deceptively hard.

Guess 3:
Radeon 9600 Pro / Radeon 9500 Pro?

That benchmark suite is pretty universal (even if I use only 3 of them ). SpecPerf is used by a lot of people too.

Seems like I should start running some more "standard" benchmarks just so I can get records in your database.

 

[Wolf]

So everytime a new benchmark or review comes out you plan to update it manually? There are tons of cards and benchmarks out there it will be a full time job keeping this "tool" updated. This seems like a lot of work for not much effect to me. I understand what you are trying to do but I don't understand how you plan on logging so many stats. I don't think you need this much of an indepth statistical analysis to figure out whats going on. Of course, I hated stats in college so maybe this is just my way of say bah it won't work well.

 

[AzBat]

So everytime a new benchmark or review comes out you plan to update it manually? There are tons of cards and benchmarks out there it will be a full time job keeping this "tool" updated. This seems like a lot of work for not much effect to me. I understand what you are trying to do but I don't understand how you plan on logging so many stats. I don't think you need this much of an indepth statistical analysis to figure out whats going on. Of course, I hated stats in college so maybe this is just my way of say bah it won't work well.

If it's like I'm thinking, then yes, he will need to manually enter the benchmark data. I don't see it being useless. In fact, it can be VERY valuable data. I sent Nite_Hawk a PM about it, but he has yet to respond.

Once the database is ready there's some cool stuff you could do with it. For instance, I had an idea last year similar to his that would create another 3D Card Buyer's Guide similar to one that Joe DeFuria created for my Dimension3D site. However, it would use actual benchmark data and the user machine info as a way to make hardware purchasing decisions. Similar to MadOnion's Performance Analyzer, but instead of solely using 3DMark scores sent in by users it would give performance metrics from actual games either from in-house testing or publicly available benchmark data.

If Nite_Hawk would played his cards right he might be able to create a standard data file template that reviewers could use to send him the benchmark data via email or his database can grab it off their servers. That way it's automatically entered into his database without having manually do it himself.

Anyway, he could do the same kind of 3D Buyer's Guide with this kind of database. For instance, the user could say he wants to buy a graphics card that plays game X the best, but he has Y kind of system. He can then even say I want to use Z kind of settings in the game. The database then could spit back a list of the top performing cards using that info. He could then click on each card to see the performance numbers he would receive or even visit the review itself. And if he wants to really make some money he can even provide a Shopper.com list of retailers selling the item.

Hopefully that gives you an idea of what could possibly be done. If I knew how to program databases and such I would have already done this. As it is, it will just have to be a dream that I have.

Tommy McClain