There's been a lot of discussion about the correct way to benchmark, which got me thinking the following.
The ideal thing you want is that, you have a particular set of games/applications that you want to run, you have a set of settings that you'd be happy with, you have a particular system environment, and you want to find out whether a video card would give a satisfactory level of performance and image quality. Of course this is impossible to do in an article, but probably not impossible to make a database for. You could do it one of two ways.
The first option is nothing new. You just put up a database online and let people submit their settings + results much like the way they submit 3DMark scores.
The second option, I think, is more intriguing. You can load a machine or a cluster of machines up with all the popular applications that don't crash. Have a web based front end that takes requests from your readers for particular applications/settings/and system environments. The front end then uses a back end scripting system to generate the appropriate settings, run the tests on the requested applications, and then send the results to the reader via email and archive the results in a database.
If you just want to change application features, it's probably not hard to do with Python or Perl just by writing configuration files and doing a system call on the appropriate executables. Heck, it's probably even possible to change the video driver settings and underclock the GPU so on the same box, you can have an Athlon 3400 emulating an Athlon 3000 and a 6800 Ultra emulating a regular 6800.
Overall, the system will operate in the following manner. JoeB3D wants to run DOOM3 at 1024x768 4xAA 8xAF trilinear filtered everything set to max on his Athlon 3000. He wants to know whether a NV40 non-ultra will do the job running the ARB path. He also wants to know how it looks. So, he goes on B3D's nifty new automatic benchmark bot, selects the settings, and types in his email. After a search of the database, the B3Dbot finds no matching existing results. Therefore, it uses a Python script to write out the appropriate autoexec.cfg file for DOOM3, sets the approriate video card settings, underclocks the cpu from Athlon 3400 speeds to Athlon 3000 speeds, underclocks the NV40 ultra to non-ultra speeds, runs the tests, records a graph of the applications and some screen shots, archives the whole thing in a database and then sends the info to Joe, who is rightfully awed at B3D's l337N355.
The whole thing will take quite some effort to set up, with possibly vexing issues with stability, the ability to run games in script mode, and the ability to change system settings. However, once you do, you'll have a unique and kick ass resource for consumers looking to upgrade hardware.
I don't think it would be unreasonable to charge a small fee through PayPal every time somebody uses such a service. You can even make it a little bit more expensive to request a previously untested configuration in order to reflect the added maintanence/wear and tear cost associated with doing the test.
What do you guys think?
The ideal thing you want is that, you have a particular set of games/applications that you want to run, you have a set of settings that you'd be happy with, you have a particular system environment, and you want to find out whether a video card would give a satisfactory level of performance and image quality. Of course this is impossible to do in an article, but probably not impossible to make a database for. You could do it one of two ways.
The first option is nothing new. You just put up a database online and let people submit their settings + results much like the way they submit 3DMark scores.
The second option, I think, is more intriguing. You can load a machine or a cluster of machines up with all the popular applications that don't crash. Have a web based front end that takes requests from your readers for particular applications/settings/and system environments. The front end then uses a back end scripting system to generate the appropriate settings, run the tests on the requested applications, and then send the results to the reader via email and archive the results in a database.
If you just want to change application features, it's probably not hard to do with Python or Perl just by writing configuration files and doing a system call on the appropriate executables. Heck, it's probably even possible to change the video driver settings and underclock the GPU so on the same box, you can have an Athlon 3400 emulating an Athlon 3000 and a 6800 Ultra emulating a regular 6800.
Overall, the system will operate in the following manner. JoeB3D wants to run DOOM3 at 1024x768 4xAA 8xAF trilinear filtered everything set to max on his Athlon 3000. He wants to know whether a NV40 non-ultra will do the job running the ARB path. He also wants to know how it looks. So, he goes on B3D's nifty new automatic benchmark bot, selects the settings, and types in his email. After a search of the database, the B3Dbot finds no matching existing results. Therefore, it uses a Python script to write out the appropriate autoexec.cfg file for DOOM3, sets the approriate video card settings, underclocks the cpu from Athlon 3400 speeds to Athlon 3000 speeds, underclocks the NV40 ultra to non-ultra speeds, runs the tests, records a graph of the applications and some screen shots, archives the whole thing in a database and then sends the info to Joe, who is rightfully awed at B3D's l337N355.
The whole thing will take quite some effort to set up, with possibly vexing issues with stability, the ability to run games in script mode, and the ability to change system settings. However, once you do, you'll have a unique and kick ass resource for consumers looking to upgrade hardware.
I don't think it would be unreasonable to charge a small fee through PayPal every time somebody uses such a service. You can even make it a little bit more expensive to request a previously untested configuration in order to reflect the added maintanence/wear and tear cost associated with doing the test.
What do you guys think?
