It was only a matter of time

[digitalwanderer]

I've read the initial white paper, and I THOUGHT I had understood the purpose...right up through the denouncing of nVidia cheats...but recent events leave me perplexed.

Sums it all up rather well right there.

 

[AzBat]

]

We also have our still ongoing effort to define what driver optimizations could be considered acceptable.

I think it 's the problem. Could you explain please why FM thinks any optimization is acceptable for a benchmark ?

I agree. NO OPTIMIZATIONS FOR A BENCHMARK should be allowed.

Out of curiosity, what about all games with built in benchmarks?

Worm,

Just answer the damn question please. We're not talking about games here, we're talking about 3DMark. 3DMark is not a game, but solely a synthetic benchmark. Please, answer the question without sending your response through some kind of lawyer interpreter.

Tommy McClain

 

[Reverend]

Once and for all, can Futuremark make a clear and concise statement or set of statements that defines:

1) What the purpose of 3DMark is supposed to be. For example
a) Representative of a guess on absolute performance with DX8 and DX9 games
b) A "stress test" for GPUs, not designed to be indicative of absolute performance levels, but an indiciation of relative GPU power in given circumstances....and what the circumstances are designed to represent.

2) Qualify exactly what 3DMark "score" represents in your view.
a) Measurement of which GPU is "faster"
b) Measurement of Which GPU is "better", (and what, qualitatively speaking, defines "better")

In my opinion, the real problem with 3Dmark, is what appears to me to be an inconsistent presentation from FutureMark of what exactly 3DMark is, how it's supposed to be used, and what the score represents.

I really have no clear indication at this time WHAT exactly the 3DMark score is trying to convey. And until FutureMark can make a CLEAR and CONCISE statement about what the PURPOSE of the benchmark is, it is near impossible to discuss on whether or not it meets those objectives, or what is and is not "acceptable" in terms of optimizations, etc.

And yes, I've read the initial white paper, and I THOUGHT I had understood the purpose...right up through the denouncing of nVidia cheats...but recent events leave me perplexed.

Joe, remove "The Gamers Benchmark" slogan and everything should be pretty much clearer. The smart folks really know what 3DMarkXX means.

 

[Joe DeFuria]

Joe, remove "The Gamers Benchmark" slogan and everything should be pretty much clearer. The smart folks really know what 3DMarkXX means.

Rev, I agree that everything is pretty clear from the whitepaper. I wholly supported it and had high praise for it actually. I spent many hours on this very message baord explaining it to those who didn't "get it."

That doesn't change the fact that what's clear to me in the white-paper, doesn't jive IMO with the allowing for "optimizations" and such.

In any case, I would really like to hear some more subjective and direct answers to the questions I posed, because again, given recent events I am not convinced that what was said in the white-paper, still holds true today.

 

[jb]

]Out of curiosity, what about all games with built in benchmarks?

Worm,

thanks for taking the time to respond. But I think I made my point clear this was in regards to 3dmark. For games. Then I agree with DT. So long as NO IQ change is seen then its acceptable. However targeting only the big benchmark games and only optimizing the standard time demos is very wrong. I hope you can agree to that.

However a benchmark like 3dmark is giving the video card a set order, a set task and a SET amount of work. If an IHV optimizes then you change the amount of work and that's kills any base line you can make for a common comparisons. To make a valid comparison you need a solid base line. If some company says games don't do it that way. Fine, then USE THAT GAME to test as well. But don't allow optimizations just because game X does. In fact we have seen from Q3 that games based of its engine and video cards FPS scores don't correlate at all (ie look at any review with Q3 and one of its other games based off it and you will see wildly different results). EVERY game out there is going to be different. If you make the change now and allow any optimizations then you have defeated in making a true benchmark.

 

[fallguy]

If FM would only let WHQL drivers in their database, it would sure hurt nVidia. They said something about that months ago, I guess they were convinced not to do it.

 

[PatrickL]

Whql does nothing to prevent cheats in drivers.

 

[fallguy]

Most of the time it seems, the cheats are mainly in the leaked drivers. The top scores are all with leaked drivers.

 

[Myrmecophagavir]

]

AFAIK There have been issues with certain chipsets reporting as "the same product" to the Database. In other words, a Radeon 9800 and a Radeon 9800 Pro might both be reporting and grouping themselves as "9800". Same with nVidia cards.

This might be a re-vamp to try and actually separate differnt SKUs of the same chip? So you can distinguish a GeForce 5900 Ultra from a 5900, or a ATI 9500 from a 9500 Pro, etc.

That was already done in the "old" HoF. We already got the (for example) Radeon 9800 and Radeon 9800 Pro, 9500 series and 9700 series all separated etc.

But it obviously didn't work for all cards, did it. The 9500 was there in the DX9 list, but if the 9500 Pro was listed separately it was out of the top 10 :? Same for the 9000. And the 9600, 9100, 8500 and 7500 were just "Series". If you're going to do it, you have to be thorough. If ATI now has SE variants going around too, will those be separately indexed? And what are you going to do about bus-width-crippled versions of cards, if they're not labelled nicely?

And please tell me you'll actually look at the results before publishing them this time and do a sanity check, to prevent things like the GF2 Ultra's former position...

 

[Chris123234]

And please tell me you'll actually look at the results before publishing them this time and do a sanity check, to prevent things like the GF2 Ultra's former position...

What happend with the GF2 Ultra's former position?

 

[Reverend]

Joe, remove "The Gamers Benchmark" slogan and everything should be pretty much clearer. The smart folks really know what 3DMarkXX means.

Rev, I agree that everything is pretty clear from the whitepaper. I wholly supported it and had high praise for it actually. I spent many hours on this very message baord explaining it to those who didn't "get it."

That doesn't change the fact that what's clear to me in the white-paper, doesn't jive IMO with the allowing for "optimizations" and such.

In any case, I would really like to hear some more subjective and direct answers to the questions I posed, because again, given recent events I am not convinced that what was said in the white-paper, still holds true today.

Like Patric had said (perhaps a couple of times) in our forums, FM is working on determining the considerations for both "absolutely no optimizations" as well as "governable optimizations" (both quoted phrases are my own words). I doubt FM personnels would explain things further regarding this when the subject matter is a WIP at FM currently.

PS. My own official (as a beta member) contribution/submission-of-opinion to the subject matter more or less agrees with what FM already has in mind, at least that's what Tero told me. I gave my answers the day after I received the doc from FM... I already know what I want in the next 3DMark.

 

[micron]

I already know what I want in the next 3DMark

I look forward to seeing the next 3DMark, I have had alot of fun with all of the versions....

 

[Nick[FM]]

But it obviously didn't work for all cards, did it. The 9500 was there in the DX9 list, but if the 9500 Pro was listed separately it was out of the top 10 :? Same for the 9000. And the 9600, 9100, 8500 and 7500 were just "Series". If you're going to do it, you have to be thorough. If ATI now has SE variants going around too, will those be separately indexed? And what are you going to do about bus-width-crippled versions of cards, if they're not labelled nicely?

We separated all cards we could. If you check the inf file of any displ driver, you should be able to see what is possible and what is not. AFAIK the Radeon9600 & Pro are listed only as "RADEON 9600 SERIES". Also, the 9500 Pro is combined with the 9700 as "RADEON 9500 PRO / 9700". There's no way to detect which is which if the driver doesn't tell us. Take a peek at the driver inf files and you will see what I mean. As soon as they separate them in the drivers, we can separate them in our online services. We are constantly updating as new drivers are released. Hope this shared some light on the subject?

And please tell me you'll actually look at the results before publishing them this time and do a sanity check, to prevent things like the GF2 Ultra's former position...

We always take a look at the lists before they are posted. Keep in mind that the lists are based on user submitted results. We don't "make" the lists ourselves. They are generated based on the data we have. But now the lists are down due to maintenance. Let's hope that all your worries will be gone when the new and improved lists will be up!

 

[digitalwanderer]

]
We always take a look at the lists before they are posted. Keep in mind that the lists are based on user submitted results. We don't "make" the lists ourselves. They are generated based on the data we have.

Thanks for taking the time to post up here, but are you saying that the list is manually made or is it just a generated list...for some reason I'm confused. (Sorry, long night. )

 

[Nick[FM]]

Thanks for taking the time to post up here, but are you saying that the list is manually made or is it just a generated list...for some reason I'm confused. (Sorry, long night. )

The list is generated. It is not made manually. We only take a look at (sort of preview) it before it goes online.

 

[just me]

Glad to hear FM is 'fixing' the HoF, just wish they'd have said the same things in my post(s) @ FM about these issues. Good thing I come here to find out what is happening on the site I frequent the most > FM.

Rev, interesting comments. Maybe an intellectual debate betrween you & Neeyik on "The Gamers Benchmark" claim. I would find it very interesting & informative.

So I guess as it stands today (Aug 22, '03) 3DM03 usage, the HoF, etc. is 'undecided'? I read Patrics' comments in his Forum about moving forward, but w/the future soo unclear, how can we? Maybe once FM makes a decision (& sticks to it this time) we'll have a guide to lead us forward. I can only hope. 8)

.02,

 

[Nick[FM]]

Glad to hear FM is 'fixing' the HoF, just wish they'd have said the same things in my post(s) @ FM about these issues. Good thing I come here to find out what is happening on the site I frequent the most > FM.

I simply answer questions here (the ones I can answer). Same thing at our own boards. The problem is that our boards are pretty huge, and therefore some questions might slip thru unanswered. If you ever need to know something important or want an answer quickly, you can always send me a PM over at our boards (or here).

 

[digitalwanderer]

]

Thanks for taking the time to post up here, but are you saying that the list is manually made or is it just a generated list...for some reason I'm confused. (Sorry, long night. )

The list is generated. It is not made manually. We only take a look at (sort of preview) it before it goes online.

Ah, you do have a sanity check then. Thank you.

 

[WaltC]

...Like Patric had said (perhaps a couple of times) in our forums, FM is working on determining the considerations for both "absolutely no optimizations" as well as "governable optimizations" (both quoted phrases are my own words). I doubt FM personnels would explain things further regarding this when the subject matter is a WIP at FM currently.

The problem is that FM has already made this decision, and explained it, in their original Audit Report .pdf in which they stated their case very persuasively.

If we apply the prinicple of Occam's Razor here (pardon my spelling if it's off), FM got it absolutely right the first time. There shouldn't be *any* benchmark-specific optimization going on, because 3dMk03 is a benchmark, not a 3d game--it doesn't even pretend to be a game. Hence, all of nVidia's optimization comments relating to 3d games simply do not apply.

The simplest, easiest, and best--completely fair--approach for FM is to tell the IHVs that no 3dmk03-specific optimizations (and certainly cheats) will be tolerated. Period. This solves the problem of undue complexity and places everybody on the same level playing field. There simply is no reason whatever to rethink their original audit report--at least, any reason having to do with the creation of an impartial benchmark. IMO, the problem here is one inherent in FM's business model, not in logic.

 

[digitalwanderer]

There simply is no reason whatever to rethink their original audit report

Well, 'cept the almighty dollar...