Like most topics at Beyond3D, this one seems to be made by a bunch of non gamers(or very casual ones) about non gaming situations.
Every time a new version of 3DMark comes out there is a group of vocal opponents who insist that it is no way represenative of actual gaming situations and as such shouldn't be used. We hear analysis and observations from numerous sources stating how the tests in no way show what future games are going to be like nor how the cards will stack up, and yet with every new release looking in hindsight the bench nearly always gives you a very clear picture of how boards will stack up in future games.
What we always end up with is a bunch of people who want to test 'current' games and show those results, and by 'current' they tend to mean a year or two old at least. The most recent title we are seeing used for benching purposes is UT2K3, a game that is ~six months old already and is held up as the best evidence of what 'future' games will perform like on given cards. I think that nearly every site should really consider a few things in terms of benching. Not so much here, as this audience tends to be far more interested in theoretical numbers(which makes sense as the audience can digest the info and figure out roughly where boards will stack up), but for sites such as H, Anand's, Tom's et al, they should really consider who it is they are writing for.
Who buys add in graphics cards?
The first response to this is not very many people. Then you have a few different categories. 3D professionals(CGI/CAD/MCAD) whom the major sites pay no real attention to(rightly so as they are such a small niche) and are rather irrelevant on this particular point.
Then you have hardware enthusiasts, whom nearly every tech site on the web targets their reviews at unfortunately. The people who want bragging rights, want to have the fastest rig they can possibly build budget allowing. Rarely(once a month or less is the norm) they will actually buy a game. They tend to spend significantly more on hardware then they do on games or applications that will actually utilize it. It is easy to understand how the people who write for the major sites can relate to these people the easiest, they go through the same motions of constantly upgrading their computers although they do so due to their line of work which tends to be the end of the spectrum they enjoy. Unfortunately, these people also tend to be the onest that wouldn't consider keeping a vid card for over a year in their rig, why would anyone mind dropping $300-$1000 a year, every year, on hardware upgrades? It shouldn't really surprise anyone that these people tend to focus their reviews on the smallest niche of purchasers of add in boards. Unfortunately for FM's crediblity, a lot of these same people use their bench as the ruler to measure their silicon penis.
Then you have the overwhelming majority, the people that aren't represented by any of the mainstream sites, actual gamers. People who somewhat grudgingly hand over six or eight new games worth of money because they have to upgrade. These people tend to move six or more 'GPU' generations at a time. This is where the real market is, the people who never register for a hardware sites forums(as a general example, AT has 2Million unique monthly vistors and ~100K registered forum users, a 20:1 disparity). These people tend to hang out in gaming forums if they spend too much time in discussion forums at all. Most of them don't want to look like a moron and will stick to their gaming based forums when they have a question.
These are the people that come to sites looking for the best purchasing decission and so often are led to a conclusion that doesn't match their needs due to hardware reviewers basing their observations on what their needs are. Their needs overwhelmingly are what ever plays 'today's'(usually a year or more older) games the best, while the typical purchaser of an add in 3D board will see a quantum leap in today's new games no matter which board they end up going with.
This is where a bench like 3DMark03 comes in so damn handy. If you look at a system that score 6K in 03 vs one that scores 2K odds are damn near certain that that 6K rig will be playing games a whole hell of a lot faster, likely in the 3x as fast range, then the rig that scored 2K a year or two down the road, the time that purchasers in this bracket tend to care the most about.
When I read a review in hardware enthusiast mode I do tend to skip right over the 3DMark score, when I'm actually looking to purchase a piece of hardware to use in my gaming rig I pay real close attention to the scores. From a hardware enthusiast point of view I care about whether a board can push 135 or 150FPS in Quake3 running 16x12x4x8, as an actual consumer I care a lot more about the likely framerate of Quake4, a cursory glance at Q3, SeriousSam, JKII scores at most a mild interest in UT2K3 and a lot of attention directed at synthetics and other benches such as CodeCreatures. Any new high end board is going to scream in old games, any simpleton can very easily deduce as much, gamers don't need twenty pages worth of review to explain that simple point.
For the reviews here, they make no qualms about what their pretenses are, 3D technology. When sites drop the singular bench with the best track record at predicting future games and still make any claims at being a games based 3D board review they are fooling themselves and more distressingly, fooling those that don't know any better.
Image quality and features are important aspects also, although we are truly well in to the point of splitting hairs comparing the highest end boards now(not that it should be ignored). Expand IQ segments and compare per pixel differences that trained eyes pretty much exclusively catch in motion, expand benches of year plus old or older games and drop the most important aspect to most gamers and consumers.
Apologies if this rant is a bit long, I have had a bit of an epiphany as I have been recently looking to overhaul my rig and realized that I had to reread almost every review I already had read due to ignoring what really mattered when it came down to it, and then realized that nearly every site that claims to be gaming based in their reviews simply ignores what is important. How are the games I'm going to buy going to run? Yet another Quake3 bench sure as hell doesn't do me any good there, but 3DMark03 does.
I would word it like this:
