New 3DMark03 Patch 330

[Joe DeFuria]

Joe:

Going further, how would you suggest they penalize NVIDIA? Stop reviewing their cards? Penalize them 10 fps, because we all know they're cheaters?

Actually, I DO think that's a legitimate punishment: stop reviewing their cards. Do that until such time that drivers are produced that do not at least have the cheating characteristics as documented by FM. (At least, do a review using a set a drivers that produces the same results pre and post the 330 patch).

As I impliked earlier, I can SEE a case for "punishing" FutureMark if, for example, the published the cheating PDF, but did not release an accompanying patch to address those issues. Point is, FM DID release a patch.

Or at the VERY least, if no nVidia punishment is given, support of the 3DMark 330 patch, should be given, including the USE of that benchmark on nVidia cards. (In addition to whatever other tests you want.) And this isn't even so much "punishment" for nVidia, as it is "rewarding" FutureMark. That is the LEAST that should have been done.

What should NOT be done, is not only failing to "punish" nVidia, but actually punishing 3DMark instead, by casting doubt over their benchmark....when there is yet no scientific refutation of their published findings, which place the blame on nVidia.

As for punishing Futuremark--they are a benchmarking company and part of their job is ensuring the data they're collecting is accurate and free of fowl play.

Correct. So when they take measures to address this (Patch 330), they should be rewarded by having its use promoted, rather than discouraged. Why should FutureMark bother trying to track down and islolate cheats (which we all agree is a good thing, right?0, if their "reward" for exposing and fixing such things is to be put on web-sites "black list"?

Bass-ackwards, no?

They're going to get dinged when they find somebody cheating this egregiously for so long a time period, apparantly undetected and relatively easily caught.

Rigth, so again, they should be REWARDED for not only informing about the cheats, but fully documenting them. The fact that THEY are getting dinged more than the DINGERS is exactly the problem.

People will naturally be suspicious of the results until they can "prove" that they've got the problem under control. Some people more than others.

People should be MORE SUSPICIOUS of nVidia's drivers, until nVidia can prove that they've gotten THEM under control.

Russ, it's a question of relativity. In no case should nVidia NOT be punished and at the same time FutureMark IS punished.

People are still suspicious that there are more Enron's, MCI Worldcoms, etc out there even though they've been caught.

Right. Even though nVidia's been caught, people are still suspicious that others are out there cheating as well. Wouldn't it be a WISE thing to endorse a benchmark which actually takes measures against cheating?

 

[Pete]

Far be it for me to butt in on someone else's argument, but:

You know what's wrong with the last three pages of this thread? Too many words.

 

[Hellbinder]

I think some people need a Time out.

I think i may need a Dictation machine to read me all these looooooooooooooooong Back and forth Discussions. Just so i can keep track. Are at the point of Hanging Chads yet?

 

[OpenGL guy]

As for punishing Futuremark--they are a benchmarking company and part of their job is ensuring the data they're collecting is accurate and free of fowl play.

I thought "fowl play" was against beastiality laws? Of course, I guess laws do vary from country to country.

 

[RussSchultz]

As for punishing Futuremark--they are a benchmarking company and part of their job is ensuring the data they're collecting is accurate and free of fowl play.

I thought "fowl play" was against beastiality laws? Of course, I guess laws do vary from country to country.

Ha! I've started typing phonetically as of late. Sometimes things like that slip through--I'm becoming part of the internet generation apparently.

 

[RussSchultz]

Wouldn't it be a WISE thing to endorse a benchmark which actually takes measures against cheating?

Sure. Only if you thought those measures were enough. Guru3d stated he felt that new drivers that were better at cheating were on their way, so he was going to stay away from 3dmk3 until things got better.

And I think its ludicrious to think that a 3d hardware review site would stop reviewing 3d hardware cards from one of the market leaders. They make money by hits, if you don't review cards, then nobody's going to come look and you go out of business.

But regardless, this whole topic has again gone way astray of the original. It has nothing to do with me thinking that the statement (applauding the way they cheated) was what Guru3d said and meant.

Going back to the bad analogies again:

Maybe a mugging...after all, it does take "balls" to hold a gun on someone, right?

Well, mugging John Gotti or the NYC police chief would be something you just have to respect. Not because you're engaged in mugging (which is quite wrong), but the outright audaciousness of who you're mugging.


Anyways, as Pete says, there's too many words in this thread. I'll stop, because no matter what I say, all it will do is end up with more words. Probably calling me a mysoginist, or a rapist, or tibettan shape shifter or worse.

 

[OpenGL guy]

Ha! I've started typing phonetically as of late. Sometimes things like that slip through--I'm becoming part of the internet generation apparently.

I wouldn't worry about it until you start using "1337 speak" and saying "t3h r0x0r"

 

[RussSchultz]

Ha! I've started typing phonetically as of late. Sometimes things like that slip through--I'm becoming part of the internet generation apparently.

I wouldn't worry about it until you start using "1337 speak" and saying "t3h r0x0r"

-targetting toward a specific shader in 3dmark03 = teh che4t

Doh! Too late!

 

[jjayb]

Quote:
Wouldn't it be a WISE thing to endorse a benchmark which actually takes measures against cheating?


Sure. Only if you thought those measures were enough. Guru3d stated he felt that new drivers that were better at cheating were on their way, so he was going to stay away from 3dmk3 until things got better.

Show me any of the benchmarks they currently use that can't be cheated on.

 

[AzBat]

Quote:
Wouldn't it be a WISE thing to endorse a benchmark which actually takes measures against cheating?


Sure. Only if you thought those measures were enough. Guru3d stated he felt that new drivers that were better at cheating were on their way, so he was going to stay away from 3dmk3 until things got better.

Show me any of the benchmarks they currently use that can't be cheated on.

Exactly. And what would happen if Futuremark decided that all of this was not worth it and decided they would rather contract out tech demos or help Remedy develop MP2? With no 3DMark to worry about NVIDIA would then focus on cheating in the next influential benchmark. Which would probably be a game benchmark that doesn't have any tools in place to catch the cheating. To me that's an even scarier proposition. At least now we have a benchmark where the developer is actively try to keep the cheating from occurring.

Anyway, if Futuremark's response(patch and PDF) are not enough to persuade sites like Guru3D that things are better, then I don't see how 3Dmark could do anything more to fix that. I think it's up to NVIDIA to decide if they want to continually test Futuremark's quest to defend it's software from being valueless.

BTW, I would still like to read Russ' comments on Joe's arguments. It's no fun to see somebody sweep good arguments under the rug.

Tommy McClain

EDIT: removed a word

 

[RussSchultz]

Show me any of the benchmarks they currently use that can't be cheated on.

That is a problem, isn't it? Nothing is cheat proof. However, notice that Humus' mandlebrot test and PCChen's precision test are showing up: It seems like one approach to thwart cheating is to go broader in the test suite (and presumably overload the engineers who are supposed to do the targeted optimization).

Another is be extra vigilant in one benchmark, which I believe Futuremark is trying to do. Are they there yet? Some would say yes, others would say no. I'm not going to fault anybody who says either way.

 

[RussSchultz]

BTW, I would still like to read Russ' comments on Joe's arguments. It's no fun to see somebody sweep good arguments under the rug.

Which ones in particular would you like me to address?

 

[AzBat]

Which ones in particular would you like me to address?

All the ones in the message at the top of this page, excluding the last sentence which you replied to. He provided answers to all your questions. So he at least deserves the same from you. Please don't ask which ones. You can read, right?

Tommy McClain

 

[RussSchultz]

I asked which ones, because there's some on the previous page too.



But if you really want:
"They should stop reviewing NVIDIA cards"
See above reply: In short--a hardware review site not reviewing hardware is a bit silly.
"They should support the 330 version".
Well, that's a matter of opinion. They apparently don't trust it yet.*
"They're not punishing nvidia"
I really don't see that they're not punishing NVIDIA. They called em cheaters, they stated they expect to have them continue cheating in the future. I presume they'll investigate closer or be skeptical of results for a while--but they can't stop reviewing the cards as that is their life blood. I can see how people are outraged and want more punishment, but I think that there's not a whole lot else that can be done as immediate punishment.*
"Futuremark should be rewarded because they're working hard as shown by 330"
Sure, and if they keep it up, I presume Guru3d will trust 3dmk03 enough to use it again. They said so in their blog. *
"Getting dinged more than the dingers"
As I said before, if you're a benchmarking company, its part and parcel of the job to ensure no cheating. They got caught with their pants down and it might take more than simply 330 to rebuild that relationship of trust with some people. I can see people believing in Futuremark from the results of 330 and their white paper, and also I can see people having reservations that the benchmark is cheat free*
"People should be more suspicious of NVIDIA's drivers"
I agree. *

* Why are we talking about this? It has nothing to do with me interpreting Guru3d's applause clause. If I somehow said something that sparked these questions, I apologize completely.

At this point, you'll see that I've offered my opinion, and Joe's offered his counter opinion. I'm offering my opinion on what I think Guru3d thinks, and listening to Joe make arguments against Guru3d's stance doesn't change my opinion of what I think their stance is. Much of my response in this thread is restatement of my original opinion so it really doesn't do much but generate more words for me to continue.

Did that sufficiently not sweep things under the carpet?

 

[AzBat]

Did that sufficiently not sweep things under the carpet?

Quick reply before leaving to pick the kids up...

Yes, that looks a lot better. Thanks.

Tommy McClain

 

[DemoCoder]

I'll state it quietly again.... we need an open source benchmark. For one thing, it would help those of us who are analyzing the cheats do our jobs much more efficiently, since we can compile modifications easily and see how it effects the drivers and it wouldn't require every hobbyist to pay lots of $$$ to futuremark to get the source. But moreover, it is far more likely that such an open benchmark will evolve more cheat resistance, because it will be patched over time after each cheat fiasco and more eyes on the problem = more chance of a solution.

We simply cannot rely on Futuremark releasing a new benchmark once or twice a year with suped up graphics and features and some small anticheat provisions. The closed nature of the benchmark makes it really hard for website reviewers to detect cheats in the first place without jumping through alot of hoops and getting lucky.


I prefer a specmark-like suite of a microbenchmarks that test each portion of the graphics pipeline, not neccessarily visually impressive, but produce the raw figures to give us perspective. If we need an end-to-end-test-everything-game-test, we can fallback to actual game engines.

But for pure benchmarking, I think fillrate, shader, T&L, stencil, HSR, etc microtests are better.

 

[AlphaWolf]

Show me any of the benchmarks they currently use that can't be cheated on.

That is a problem, isn't it? Nothing is cheat proof. However, notice that Humus' mandlebrot test and PCChen's precision test are showing up: It seems like one approach to thwart cheating is to go broader in the test suite (and presumably overload the engineers who are supposed to do the targeted optimization).

Humus and PCChen have to be careful though, I am sure nvidia is working on a statement on how they are intentionally creating a situation to make them look bad.

 

[Hellbinder]

That is a problem, isn't it? Nothing is cheat proof. However, notice that Humus' mandlebrot test and PCChen's precision test are showing up: It seems like one approach to thwart cheating is to go broader in the test suite (and presumably overload the engineers who are supposed to do the targeted optimization).

I have thought and pushed here and there for a long time for Reviewers to just stinking use FRAPS and play a game for about 5 min, or play a complete level. Then simply show the FPS per second in a line Graph over that period. Do gamers really need this microsecond per frame speed analasys on the same Freaking 5 benchmark apps everyone uses???

I have yet to see any evidense on my own personal testing where FRAPS is not Right on the money or within 1 FPS of the in game Frame counters i have used.

Freaking log into Battlefield on an average server and see what you get after a 15 min match. Throw out the obvious Highs and lows and just show an analasys of FPS over time. Its freaking easy. Its 99% accurate and it can be used in all the games we never see tested. Like Madden 2003, Hockey games, Racing games, adventure games, Flight Sims, the Sims, hell you name it. It would give a much greater and Real world performance View of hardware than even the best analasys seen on B3D. Because wether we like it or not, its still canned benchmarking.

(with FRAPS you can save th FPS to File in continuous or every second)

 

[RussSchultz]

]
I have thought and pushed here and there for a long time for Reviewers to just stinking use FRAPS and play a game for about 5 min. Then simply show the FPS per second in a line Graph over that period.

The problem with fraps is reproduceability so that you can assure people:
a) You're treating all parties the same.
b) You're doing the testing correctly.
c) The results aren't a fluke.

Though, I agree, that would be a helpful addition to any review (but you can't use it as the cornerstone)

 

[Hellbinder]

Not if you play a complete level of a game. Heck how can anyone trust anyone?

You either Trust the Reviewers or the IHV's Which is it? I see no reason why it should not be *THE* cornerstone of every review. With the 3dmarks, and Demo1's of the world taking second place only as a proof of product.