Futuremark has problems here.

[jvd]

Why do people still bother with this benchmark.

Stop using it . Stop going to thier websites. Once enough people do they will go out of busniess and we don't have to put up with another spineless company that allows venders to produce sub par images to get better scores .


The company lost all respect from me once they backed down from calling it cheating.

 

[vb]

this is getting stupid. what is wrong with you, people? What is so hard to understand?

They screwed up!

Plain and simple. they missed the cheat in PS2.0 test. they released 340 after 52.12 to disable all cheats. they missed one. so shoot them!

Fortunately, it doesn't affect the final score. maybe that is why they missed it. That and nvidia's claim that they don't cheat theoretical tests, just game-like scenarios.

I can't believe you don't understand that. But if you don't want to understand it, then just go ahead and use 53.03 or whatever and you can be sure it will cheat like crazy.

You don't want to use 3Dmark, don't. but all this talk will do is stop futuremark from revealing any other cheats they find. This is work in progress!

As for me, I prefer 3dmark 340 as it was last year then perfect next year. Maybe it is just me.

 

[andypski]

Why do people still bother with this benchmark.

Stop using it . Stop going to thier websites. Once enough people do they will go out of busniess and we don't have to put up with another spineless company that allows venders to produce sub par images to get better scores .

The company lost all respect from me once they backed down from calling it cheating.

So, you appear to be advocating shunning a benchmark company that is at least attempting to do something about unwanted manipulation of their benchmark in favour of the other totally unregulated benchmarks out there where absolutely nothing is done by the creators to enforce correct driver behaviour? After all, at the moment that is the choice that you have, is it not?

That is not a solution.

If you put out of business companies that are trying to do something about benchmark manipulation then you only succeed in creating precisely the situation that someone who wishes to cheat in benchmarks will enjoy - ie. that benchmarks become a free for all with no enforcement of appropriate standards.

If you dislike how they are dealing with the situation then fine, but the reason that people should still bother with the benchmark is abundantly clear - pretty much any attempt to enforce correct benchmarking guidelines is better than making no attempt at all.

Why are you advocating support for unregulated benchmarks over the one benchmark that has some level of regulation?

 

[andypski]

Well according to Futuremark it's the reviewers fault for not being aware of their driver policy.

That sounds like a perfectly reasonable position to me.

If we as an industry want to prevent the manipulation of benchmark results then it has to be done by collaboration between reviewers and the people who create the benchmarks. This collaboration has to include responsibility on the part of the reviewers to research the most current guidelines on the correct way to run a given benchmark application.

Otherwise it's like burning your mouth on a fruit pie despite the fact that the packaging says "Caution, filling is hot" all over the place, and then suing the makers of the pie because you couldn't be bothered to read the warnings.

The result of this is that you will only get cold fruit pies with no warnings in future, that way you won't be able to hold the makers responsible for burning your mouth. They'll be crappy fruit pies though.

 

[Nick[FM]]

You say that FM does not allow ANY 3DMark specific optimizations for any of the tests? Then you say that "the theoretical PS2.0 score is solely comparable between NVIDIA cards (when using the 52.16 drivers)."

So it seams to me that there are optimizations in the APPROVED FM drivers (52.16).

On the surface this leaves me with two thoughts. One is you are lying, as in something a crapy politician would do. The second is you don't know what you are talking about. I doubt the second.

So please explain why I should not think you are lying.

I am not lying, nor am I clueless what I am talking about. The theoretical PS2.0 test does not affect the 3DMark result in any way. As we noted on the approved page, the theoretical PS2.0 test is only comparable between NVIDIA cards, when/if you are using the 52.16 drivers. The 52.16 drivers are approved to be used with 3DMark03 build 340, but with the note about the theroretical PS2.0 test.

OMG I read the thread. If the FM members that posted in that link represent the offical FM stand then FUCK them. FM has taken a bad situation and made it worse. I hope FM employee's are looking for a different job to get away from that POS company.

Please note that only me and a user named Captain are the only admins/mods who are Futuremark employees. The others are users, doing moderator & admin duties voluntarely.

If not , why could Forceware 52.16 be approved by Futuremark?
As you said, Forceware 52.16 still have 3DMark specific optimizations.

They are approved, with the exception of the theoretical PS2.0 test, which score is (as said a couple of times) solely comparable between NVIDIA cards.

Well according to Futuremark it's the reviewers fault for not being aware of their driver policy.

No, this is not really the case. We have tried to inform the media about the 3DMark03 usage guide & approved drivers, but unfortunately some sites haven't got the information. I personally do not blame anyone. To inform sites about things like these is pretty difficult as there are maybe more than 200 sites posting reviews and new ones are coming every day. It is not that easy to keep track on all websites, and which ones are using 3DMark03 in their benchmark suite. Of course we add more and more websites to our press mailing list, and hopefully we have a better coverage next time. Sorry if you got the wrong picture!

*edit: It seems that very few people read the EULA. There it is very clearly stated (in the very beginning of it) that "PLEASE NOTE that Section 11 of this Agreement contains terms, which restrict Your right to publish any Results obtained by You using drivers that do not fulfil Futuremark Optimization Guidelines requirements. Updated Optimization Guidelines and list of drivers verified by Futuremark can be found on Futuremark’s website at www.futuremark.com."

 

[Deathlike2]

*edit: It seems that very few people read the EULA. There it is very clearly stated (in the very beginning of it) that "PLEASE NOTE that Section 11 of this Agreement contains terms, which restrict Your right to publish any Results obtained by You using drivers that do not fulfil Futuremark Optimization Guidelines requirements. Updated Optimization Guidelines and list of drivers verified by Futuremark can be found on Futuremark’s website at www.futuremark.com."

That's obvious as it stands. If there were more "notifications" in the software (eg. after a 3DMark run, you send out a popup box showing the reminder), then this wouldn't happen. It's not everyday that you see people read the EULA when installing Windows (any version) at any given time.

Honestly though, if 3DMark had some sort of "application update" mechanism (to update the program effectively) or "approved drivers list update" built into the program (not everyone is going to visit the Futuremark website to check up on this list), this would help Futuremark greatly.

When you have such a feature, you could easily give a popup saying, "This driver version (bleh) may not have followed the Futuremark guidelines. Check back in a couple of days to determine a change in its status. If this doesn't change in a few days, you can assume they do not following our guidelines."

 

[digitalwanderer]

Slightly OT, mebbe not....I just wanted to say "thanks" to Worm for stopping by and posting here on this.

Thanks Worm, it's appreciated. It's good to know FM cares.

 

[kyleb]

remark]"

If not , why could Forceware 52.16 be approved by Futuremark?
As you said, Forceware 52.16 still have 3DMark specific optimizations.

They are approved, with the exception of the theoretical PS2.0 test, which score is (as said a couple of times) solely comparable between NVIDIA cards.

if you say they are solely comprable between nvidia cards then how come i can have my radeon set as my current project and click on a compare link of someone's that is useing forceware 52.16, only to find myself comparing the ps2.0 tests scores? if the scores are not comprable why does futuremark help me compare them, isn't that hypocritcal?

 

[Tridam]

]I am not lying, nor am I clueless what I am talking about. The theoretical PS2.0 test does not affect the 3DMark result in any way. As we noted on the approved page, the theoretical PS2.0 test is only comparable between NVIDIA cards, when/if you are using the 52.16 drivers.

Can you state that PS2.0 test is comparable between a GeForce FX 5800 and a GeForce FX 5900 ? Or between a GeForce FX 5600 and a GeForce FX 5700 ?

 

[ByteMe]

]
I am not lying, nor am I clueless what I am talking about. The theoretical PS2.0 test does not affect the 3DMark result in any way. As we noted on the approved page, the theoretical PS2.0 test is only comparable between NVIDIA cards, when/if you are using the 52.16 drivers. The 52.16 drivers are approved to be used with 3DMark03 build 340, but with the note about the theroretical PS2.0 test.

]
Please note that only me and a user named Captain are the only admins/mods who are Futuremark employees. The others are users, doing moderator & admin duties voluntarely.

I can understand that you might have a family to feed. I can also understand how this can make a man push down pride/ethics for a bit to survive.

Even understanding this I would NEVER do any business/hire with someone that shows these "flavor" of ethics. How could you explain this to your kids? I also don't understand why B3D would want to associate with someone/company with this type of behaivor. You are doing the graphics industry/users a great disservice.

<insert random your moma insult here>

I am done with this topic.

 

[jvd]

Why do people still bother with this benchmark.

Stop using it . Stop going to thier websites. Once enough people do they will go out of busniess and we don't have to put up with another spineless company that allows venders to produce sub par images to get better scores .

The company lost all respect from me once they backed down from calling it cheating.

So, you appear to be advocating shunning a benchmark company that is at least attempting to do something about unwanted manipulation of their benchmark in favour of the other totally unregulated benchmarks out there where absolutely nothing is done by the creators to enforce correct driver behaviour? After all, at the moment that is the choice that you have, is it not?

That is not a solution.

If you put out of business companies that are trying to do something about benchmark manipulation then you only succeed in creating precisely the situation that someone who wishes to cheat in benchmarks will enjoy - ie. that benchmarks become a free for all with no enforcement of appropriate standards.

If you dislike how they are dealing with the situation then fine, but the reason that people should still bother with the benchmark is abundantly clear - pretty much any attempt to enforce correct benchmarking guidelines is better than making no attempt at all.

Why are you advocating support for unregulated benchmarks over the one benchmark that has some level of regulation?

I'm advocating that we use benchmarks that have strictly enforced rules that the company is willing to bet its life on it .

I don't see futurmark trying to change anything . I saw them stand up and then skimper back into a corner. I saw them retract all thier statments on cheating . I saw them say that now there can't be optimizations. I've seen nvidia repeatly break the rules . I have seen futuremark say nothing just silently push out a patch that is already defeated.

Why should I support cowards ?

THey aren't fighting anything . They did just enough to try and safe face and get money from nivida .

Whats even worse is now nvidia is a partner . They are part of your test program. Which gives them even more of a chance to break the rules .

I don't know how many smart men there are at futuremark but I would never go into busniess with a person or company that has dragged my product into the mud and made it loose any meaning it once had .

 

[Quitch]

While FutureMark may not be making a great job of it, please point us towards the game/benchmark that is doing a better one?

I find it amusing when people dump 3DMark03 but continue to use Aquamark.

 

[Nick[FM]]

I am done with this topic.

Ok, though I still don't see what is wrong with my(?) ethics here. I am simply supporting & standing behind our products to 100% and trying to make the whole situation better for everyone. If you think that is wrong or a bad thing, then .. well, what can I say.

 

[engall]

]

I am done with this topic.

Ok, though I still don't see what is wrong with my(?) ethics here. I am simply supporting & standing behind our products to 100% and trying to make the whole situation better for everyone. If you think that is wrong or a bad thing, then .. well, what can I say.

from Futuremark Optimization_Guidelines_QA.pdf
http://www.futuremark.com/companyinfo/Optimization_Guidelines_QA.pdf
I found

Q: How will Futuremark enforce these new rules?
A: We have internal tools and resources with which we will test new drivers against the guidelines.
We will maintain a list of drivers we have determined to be in accordance with the guidelines. We will
publish detailed description of this process including actions when violations have been discovered.

And we wanna know what about nVidia latest drivers Forceware 53.03!
If these drivers have 3dmark03 specific optimization and violations,
why not publish detailed description of this process including actions ?
And if not, why are not Forceware 53.03 approved by Futuremark?
What the hecks going on?

 

[Nick[FM]]

]

I am done with this topic.

Ok, though I still don't see what is wrong with my(?) ethics here. I am simply supporting & standing behind our products to 100% and trying to make the whole situation better for everyone. If you think that is wrong or a bad thing, then .. well, what can I say.

from Futuremark Optimization_Guidelines_QA.pdf
http://www.futuremark.com/companyinfo/Optimization_Guidelines_QA.pdf
I found

Q: How will Futuremark enforce these new rules?
A: We have internal tools and resources with which we will test new drivers against the guidelines.
We will maintain a list of drivers we have determined to be in accordance with the guidelines. We will
publish detailed description of this process including actions when violations have been discovered.

And we wanna know what about nVidia latest drivers Forceware 53.03!
If these drivers have 3dmark03 specific optimization and violations,
why not publish detailed description of this process including actions ?
And if not, why are not Forceware 53.03 approved by Futuremark?
What the hecks going on?

The Q&A you quoted is refering to this PDF:

http://www.futuremark.com/companyinfo/Enforcement_Process.pdf

and the process & actions described in it.

 

[digitalwanderer]

We will maintain a list of drivers we have determined to be in accordance with the guidelines. We will
publish detailed description of this process including actions when violations have been discovered.

Uhm, Worm? You didn't really answer the man's question, you either misunderstood or side-stepped it.

He wanted to know what specifically you didn't approve of in the 53.03 set I believe, as well as what actions you plan to take. (See? Your own statement actually implies that you're going to take more action than just not posting the drivers to the approved list. )

 

[Joe DeFuria]

If we as an industry want to prevent the manipulation of benchmark results then it has to be done by collaboration between reviewers and the people who create the benchmarks.

I agree 100%.

My specific issue with Futuremark and how they're handling this, is that it's NOT a collaboration the way I look at it.

They are forcing reviewers to install older driver versions in order to get "valid" scores. As I predicted earlier, this seems to be causing a few things to happen:

1) Some web sites just ignore it, and run and publish with unapproved drivers anyway
2) Some web sites are flat out refusing to use 3DMark, because of the "hassle."

I want to be perfectly clear about my position on this. I want FutureMarl / 3DMark to succeed. I do agree that they are doing at least SOMETHING to combat the rampant cheating by IHVs in benchmarks, and if the Futuremark guidelines are in fact adhered to, then that would be great. And ultimately, I place the blame for this whole mess on nvidia. But Nvidia, no matter how much we (end users) bitch and moan at them, isn't going to change.

So, FutureMark should properly collaborate with reviewers, such that a solution is reached that reviewers can live with. Again, IMO, this means that FutureMark should release patches for 3DMark (when needed) after each driver release, to defeat detection / cheating.

It's the obligation of the reviewers to download the latest 3DMark patch, just as they are "obligated" to download the latest drivers from the IHVs.

Of course, this is more work for FutureMark. But I feel it is the ONLY REASONABLE WAY that they can properly enforce their guidelines.

Without doing this, I just see more and more review sites not bothering with it...which again is a shame because it is a great tool.

Otherwise it's like burning your mouth on a fruit pie despite the fact that the packaging says "Caution, filling is hot" all over the place, and then suing the makers of the pie because you couldn't be bothered to read the warnings.

I see the analogy as a little different.

It's like the pie makers put a warning label: "Caution! Filling is Hot....and in order not to burn yourself, you must place this in the freezer first for 10 minutes to cool it just enough to reach validated eating temperature."

So, the "eater" has a choice to make, 1 of 3 ways to go:

1) Go out of his way to properly following the label, put it in the freezer for 10 minutes and get the "legitimate" results....
2) Ignore the warning and eat it "very hot" anyway, because it's too much work or takes too much time to put it in the freezer, or because he THINKS it's better "really hot."
3) Decide he just can't be bothered with this brand of pie, even if it gives superior results when the directions are followed...and just chooses a different pie without all these cooling requirements...

I'm betting on 2 or 3 happening way more often than one.

The result is STILL that you will only get cold fruit pies with no warnings on them...because no one buys enough hot pies to sustain the company and keep them in business. (Note that when the pie maker finds out that you went with decision 2...they come after you and slap you on the wrist.)

 

[engall]

We will maintain a list of drivers we have determined to be in accordance with the guidelines. We will
publish detailed description of this process including actions when violations have been discovered.

Uhm, Worm? You didn't really answer the man's question, you either misunderstood or side-stepped it.

He wanted to know what specifically you didn't approve of in the 53.03 set I believe, as well as what actions you plan to take. (See? Your own statement actually implies that you're going to take more action than just not posting the drivers to the approved list. )

Yes,I wanna know what specifically you didn't approve of in the 53.03 set and what actions you plan to take.You got the point.
Thank you for your input.

 

[Mark]

Is it technically too difficult/impossible for Futuremark to implement a sort of anti-detect mode, like in ShaderMark2? Of course, Futuremark would still need to make sure that IHVs like nVidia aren't defeating the anti-detect scheme...

 

[Quitch]

I'm with Joe. If you're not running the latest version then users will be up in arms, but break some guidelines... meh, most users won't even have heard of them.