Legit Reviews cops - out on 3DMARK

[nelg]

LEGIT REVIEWS Editorial

We always want to give significant and accurate results to our readers. We have been continuing to include 3DMark03 scores because we thought it was a benchmark that many still thought was significant despite nVidia's optimizations. We now realize that due to nVidia's actions, 3DMark03 scores are insignificant and inaccurate. We can not change this. Only FutureMark and/or nVidia can bring this benchmark back to life.

We also always use the latest drivers available from nVidia's website for our tests. So far, FutureMark's solution is to only allow users to publish 3DMark03 scores using approved drivers. We do not think is a good idea because a newer version of a graphics driver could potentially fix a bug that is not 3DMark03 dependent, but would still slightly affect its score. Now that is not currently the case, but it could be. So in order to maintain a Standard Operating Procedure, we feel it is best to NOT include 3DMark03 in our graphics solution reviews any longer.

So as of today, January 26, 2004, we will no longer include 3DMark03 in our graphics solution reviews in order to use the most recent graphics driver in all of our tests.

What I find hard to swallow is that they recognize that nVidia is cheating yet they respond in such a way as to reward the cheating .The very fact that nV cheats raises a series of very important questions. Questions like why, how much of a difference does it make, are such tricks needed for games (if possible). I understand that it is more work to have to use two different sets of drivers to use 3DMark but if you believe in the validity of synthetics then do the extra work. There is a large chorus of people suggesting things that FutureMark could do to help rectify the situation, some of which are valid, but what about the question of what reviewers could do? Is it really to much to ask to install a different set of drivers?

 

[Tim Murray]

Seems silly. Global bugfixes that affect performance in a year-old architecture? I doubt it.

 

[3dilettante]

It looks to me that 3dmark 2k3 is in a no-win situation.

The corporate situation reeks of apparent conflict of interest, and the small futuremark corporation has absolutely no traction against a big player like Nvidia that can and does easily toss it around like an orca toying with a seal (for some reason, that is what comes to mind).

The benchmark itself has lost a lot of its importance. The game tests, where futuremark has spent most of its energy trying to prevent cheating, are probably the tests that became unimportant in the least amount of time. There are now games that have some DX9 capability, and their usage, even if imperfect as benchmarks, probably are better metrics for performance than the nearly entirely GPU-bound game tests.

When there weren't any such games, the game tests were somewhat useful predictions, but now they are about as useful as last week's three day weather forecast.

That there are still cheats in non-scoring tests also creates doubt about any attempt at trying to test various card capabilities in isolation, as opposed to the more amorphous nature of the game tests. At least with theoretical tests that test features separately there is a useful amount of information to be found, and regardless of the vagaries of the gaming scene, such tests maintain a baseline of utility. Not that there is a perponderance of such tests to start with.

Now, the less relevant game tests were temporarily more correct, while the other tests were for a time more correct, but less so.
The patch for 3dmark was at best treading water, and it is silly to expect the entire hardware world to have to keep a back revision of drivers for the sake of a benchmark, especially not one that really doesn't have the most meat to it.

I don't know about winners in this situation, but futuremark definitely lost, and unless it can creatively come up with a way to change the situation, it will continue to do so.

 

[Joe DeFuria]

So as of today, January 26, 2004, we will no longer include 3DMark03 in our graphics solution reviews in order to use the most recent graphics driver in all of our tests.

Well everyone (including FutureMark), I TOLD YOU SO.

 

[nelg]

So as of today, January 26, 2004, we will no longer include 3DMark03 in our graphics solution reviews in order to use the most recent graphics driver in all of our tests.

Joe, I didn’t write that. That was from Legit Reviews. Most of what I do is illegitimate.

 

[digitalwanderer]

Well isn't that a fine howdy-do after all the help we gave him on figuring out how to use 3dm2k3 properly, he abandons it.

Oh yeah, Joe? You're right, you did call it...I'm just hoping the Futuremark does something about it.

 

[Joe DeFuria]

Joe, I didn’t write that. That was from Legit Reviews. Most of what I do is illegitimate.

Hahah...yes, I know you didn't write it...sorry about that...I'll sort out the quote...

 

[Richthofen]

who cares about Futuremark.
We don't need that crapmark anyway.

I wanna have games benchmarked and not synthetic crap like 3dmark, PC mark, Sysmark or Sisoft Sandra....

 

[3dilettante]

...I'm just hoping the Futuremark does something about it.

I'm sure FutureMark has a few plays left in its playbook. Let's hope for their sakes they aren't all in Chapter 11.

 

[digitalwanderer]

...I'm just hoping the Futuremark does something about it.

I'm sure FutureMark has a few plays left in its playbook. Let's hope for their sakes they aren't all in Chapter 11.

I don't even know if they know they have a problem, they seem in utter denial about it.

 

[John Reynolds]

3DMark2004 - The Partial Precision Benchmark.

This will be the case or FM will continue to see its suite fade.

 

[digitalwanderer]

3DMark2004 - The Partial Precision Benchmark.

This will be the case or FM will continue to see its suite fade.

OMG, isn't that sort of basically a modified "Let the Wookie win!" strategy?!?

 

[Dave Baumann]

Why? PP_ is something thats being used by developers in games, so why not use it in a future benchmark? If there is no decrease in quality from not running in full precision then there is no point in running in full precision as there will be cases where performance will increase without loss of quality. If its right to use partial precision then is should be used (the only reason I think it wasn't used in 03 was because it was a late addition to DX and there was no education on it when 03 was going through its final stages).

 

[digitalwanderer]

Then why doesn't FM just put out a patch for 3dm2k3 that allows PP_ hints? Wouldn't that just make nVidia cream their pants and do backflips?

Lemme guess, it ain't quite that simple.... :?

 

[Mariner]

Then why doesn't FM just put out a patch for 3dm2k3 that allows PP_ hints? Wouldn't that just make nVidia cream their pants and do backflips?

Changing all the shaders to add PP would make comparisons between old and new scores invalid which would bugger things up somewhat, I'm sure.

I wonder if NV would still feel the need to do shader replacements in a 3DMark2004 which supported PP hints? Ready for a complete 180 degree tack once again in NV's driver policy everyone?

 

[Brent]

Then why doesn't FM just put out a patch for 3dm2k3 that allows PP_ hints? Wouldn't that just make nVidia cream their pants and do backflips?

Changing all the shaders to add PP would make comparisons between old and new scores invalid which would bugger things up somewhat, I'm sure.

comparison between old and new scores are already invalid because of new patches that change performance and new drivers that implement optimizations that change performance

 

[Deathlike2]

I wonder if NV would still feel the need to do shader replacements in a 3DMark2004 which supported PP hints? Ready for a complete 180 degree tack once again in NV's driver policy everyone?

NV will probably continue to do shader replacements in the full FP test.

To summarize what NVidia will say, here I go:

"Games will not fully optimized without our hand-optimized shaders in our unified compilier. The extra precision is not necessary and will provide little or no degradation of IQ in games."

 

[Dave Baumann]

The new patches are reinstating the actual requested shaders.

 

[demalion]

A point of disagreement:

...
The benchmark itself has lost a lot of its importance.

I don't think so, as it was only ever useful as one targetted and informative point of data anyways. It still serves in that regard (with the right IHV/driver combination to remove cheating as a factor in certain tests).

The game tests, where futuremark has spent most of its energy trying to prevent cheating, are probably the tests that became unimportant in the least amount of time.

How do you figure? They became the least unique in what they utilized in comparison to games in the shortest amount of time, because they were aimed at representing games. This is because Futuremark was accurate in game utilization of the factors as they set out to be.
As for the other tests, there were vertex processing and pixel shading 2.0, etc., synthetics out at almost the same time, or before. I don't think uniqueness in utilization was an intended value.

There are now games that have some DX9 capability, and their usage, even if imperfect as benchmarks, probably are better metrics for performance than the nearly entirely GPU-bound game tests.

This is the heart of my disagreement here...this statement seems completely wrong. It is being intentionally GPU bound that sets 3dmark apart as a synthetic, on purpose, and makes it less imperfect as a benchmark than (almost all) games. How do you come up with less GPU bound being a better metric for GPU performance?
All it is is more indicative of that actual games performance than...that actual game, which isn't a surprise, as you are measuring that actual game. Some other game certainly isn't any better, until you dig through and somehow isolate the factors for comparison that a synthetic isolates for you. That's why 3dmark 03 was already important as one point, but not as the only point.

What you seem to be attacking is the lazy reviewer's usage who only used 3dmark and thought it was all that was needed. That's a good thing to attack, but all too often discussions of this matter persist in proposing that this usage defines 3dmark, when it clearly doesn't, and Futuremark has said as much.

When there weren't any such games, the game tests were somewhat useful predictions, but now they are about as useful as last week's three day weather forecast.

For instance, this seems to only make sense with the criteria of 3dmark being used alone, and/or ignoring how well it corresponds to measuring the hardware performance.

I agree that a failure to remove cheats in non-scoring tests is a significant problem and source of confusion, however.

 

[Deathlike2]

Then why doesn't FM just put out a patch for 3dm2k3 that allows PP_ hints? Wouldn't that just make nVidia cream their pants and do backflips?

Changing all the shaders to add PP would make comparisons between old and new scores invalid which would bugger things up somewhat, I'm sure.

comparison between old and new scores are already invalid because of new patches that change performance and new drivers that implement optimizations that change performance

I doubt that's completely true.

I'm not sure if any other venders (other than NVidia and XGI) implemented 3DMark03 optimizations... however you should test older versions 3DMark03 with the post 3.4 CATALYST driver (whatever driver version that removes all 3DMark03 optimizations) and see if there is any difference in performance between 3DMark03 versions.

If they are similar, then whatever changes in 3DMark03 do not make a difference... otherwise it isn't the case.