If you can't see it, it's not cheating.

[Evildeus]

Well martrox, if you think it's circle logic good for you, but can you point me where it is? Unfortunately, it's not because Xbt says that what Nv did is wrong, that it's good for all reviewers.

I'm not saying that Nvidia didn't cheat in my point of view, but that the determination of "cheat" depends on the rules people makes, and reviewers/benchmark company can put rules where what Nv did/is doing is not cheating, as any championship can make a rule where taking drugs is not cheating also.

Sorry if that disturbs you.

PS: I thought that the world was not black or white

 

[WaltC]

What I found so amazing and comical about the original "cheating you can't see isn't cheating" remarks was that in 3DMk when you took the camera off the track you could then see the cheating in all its glory... So the original premise of the sentiment was incorrect to begin with. That statement should have been:

"I can't see the cheating so I say, 'What cheating?'" I guess the originator of the sentiment read the ET article but never got around to looking at the screen shots--or if he did, was unable to correlate the two intellectually...

Also, "a general goal of all cheating is that it go undetected." Ethics for 1st-graders by R.M. Nixon IV, Bantam Press, 1998.

 

[Hanners]

I'm not saying that Nvidia didn't cheat in my point of view, but that the determination of "cheat" depends on the rules people makes, and reviewers/benchmark company can put rules where what Nv did/is doing is not cheating, as any championship can make a rule where taking drugs is not cheating also.

Well, FutureMark made their rules, and nVidia broke them.

Any review/benchmark that states that 'cheating is okay' is going to be about as popular as any sporting body who says 'cheating is okay'. Nobody in their right mind would do it if they want to keep their credibility intact.

 

[Chris123234]

So if I am in an Olympic game and I take drugs, I have escaped the drug tests undetected but since I am not caught taking drugs it isn't considered cheating.

Can anyone see a flaw with that?

Not insulting you but that analogy sucks. You cant use the ethics for an Olympic game and stick them to video card optimizing. Drugs would only benifit the athlete but optimizing benifits the gameplayers. That is if your talking about games and not benchmarks. If it is about benchmarks just ignore this then.

 

[Bouncing Zabaglione Bros.]

Not insulting you but that analogy sucks. You cant use the ethics for an Olympic game and stick them to video card optimizing. Drugs would only benifit the athlete but optimizing benifits the gameplayers. That is if your talking about games and not benchmarks. If it is about benchmarks just ignore this then.

There's nothing wrong with optimising games (as long as you don't alter the output). What *is* wrong is when you optimise the "benchmark mode" of those *same* games with techniques that cannot be used in-game (clip planes, no buffer clears), or that changes the output (low quality precision or low quality shaders) in order to convince people that your card is faster than it actually is when playing the game.

Those same techniques when extended to synthetic benchmarks create the same problem - they mislead the buying public by presenting performance that cannot be duplicated in games themselves.

This is why so many people are unhappy with Nvidia's recent behaviour. What they are doing *does not* benefit the gamer, but misleads gamers instead, by convincing them to buy products that do not deliver the performance that the Nvidia "optimised" benchmarks say they will.

 

[Evildeus]

I'm not saying that Nvidia didn't cheat in my point of view, but that the determination of "cheat" depends on the rules people makes, and reviewers/benchmark company can put rules where what Nv did/is doing is not cheating, as any championship can make a rule where taking drugs is not cheating also.

Well, FutureMark made their rules, and nVidia broke them.

Any review/benchmark that states that 'cheating is okay' is going to be about as popular as any sporting body who says 'cheating is okay'. Nobody in their right mind would do it if they want to keep their credibility intact.

FM said that, but they can't have enforcement on what other people think/decide to be the rule on their reviews. You can't use some cheating drivers on ORB, but you can in any review...

That's why i say, there's no cheating if you don't break a rule, and the rule is provided by the one enforcing it. FM can't stop a hardware site using the Nv drivers cheating on FM basis, but not on this hardware basis (which says if i don't see it it's not cheating).

Still, i think you are confusing between cheating and doing the same thing if there's no rule against it...

*edit* And i don't see why you all make a difference between benchmark and games.

 

[martrox]

And i don't see why you all make a difference between benchmark and games.

You don't? Why am I not suprised! Most all of us here do understand the difference.

 

[Evildeus]

Really? That's why all people are asking themself if they need to do home-made benchmarks?

There's no difference. In your way of thinking, things should be done in a way, if it isn't then it's cheating, but when it's game it's ok? Then if someone (say me), think like the 1 of the 2 following statement, there's an issue? :
- There's no difference between game and benchmark, if i don't see the optimisation, it's ok
- There's no difference between game and benchmark, if it's "optimized" then it's cheating

PS: I'm still waiting proof of my "circle logic"

 

[WaltC]

.....
Still, i think you are confusing between cheating and doing the same thing if there's no rule against it...

*edit* And i don't see why you all make a difference between benchmark and games.

How about "rules" like this: Thou shalt not dishonestly represent thy products.

E, I think this is the "rule" broken that upsets most people.

There's a major difference between benchmarks and games for several reasons:

1) It's possible to cheat benchmarks in ways you cannot cheat games

2) Benchmarks are meant to compare and investigate hardware; games are meant to be played

3) Benchmarks are designed to show average-case performance for 3D software running under an API. The great majority of games are average-case, meaning there is no application-specific, performance optimization for them at all in 3D card drivers. Therefore, a benchmark which is is either cheated or optimized for will return performance values incongruent with the product's performance in the majority of 3D software available. This means it will return a *false* performance picture.

The difference is as clear as the difference between a benchmark and a game.

 

[Tridam]

Evildeus -> You can't compare games and benchmarks. A game could be a benchmark. You have to compare synthetic benchmarks and games benchmarks. Optimising in synthetic benchmark is cheating. Global optimisation in games benchmark can be good if the quality don't suffer too much and if these optimisations are not just timedemo related.


IHVs should change to way they optimise. They have to make it optional. If they don't make it optional, the quality will incessantly go down.

One quick example: IHV1 drop a little the filtering quality. It improves performances. IHV2 with full quality has a marketing problem because performances of its product are not as good. IHV2 knows of course how to boost performances. So IHV2 will quickly drop the filtering quality to be competitive with IHV1. Some weeks after, IHV1 want to improve performances one more time. He decides to drop quality a little more… … … it's a vicious circle…

Unfortunately, IHVs don't want to be honest with optimisation because it's the only way they have to make a 'bad product' look good. They know that reviewers will use the full quality option (or show number with and without optimisations). They know that performances in full quality are very important. So they don't want to make available a full quality option if they know that their products won't look good with it. They call a medium quality mode "High quality mode". And they drop the full quality possibility (NVIDIA do exactly this. With earlier drivers it was possible to have a better quality with GeFFX than today. Now this option has gone and the quality mode has more compromises. Of course NVIDIA can say "Hey ! Here are some new drivers, we have greatly improved our full quality performances!")

It's really boring… When I buy (it's been a while ) a card which costs 300-400-500 $/€, I want to have the possibility to play with full quality. I don't want filtering tweaks. If I pay a card like that, I pay for quality ! Of course, high end cards have just a goal : help IHV selling as much full trucks of low end cards as possible (it's a little caricatured but not too much). High-end cards have not to respect one of the main rules of luxurious products : the quality. The high-end video card has just to show great numbers and have some great new techmarketing vocabulary. Right now, it's enough to help marketing teams to sell low-end parts. I hope this will change Compromises are good but they have to be optional. If IHVs don't want to have a not-good-number mode by default, they could have the honesty of making available a no compromises mode.


If I can't see an optimisation when playing a game, it's a good optimisation.
If I see a little quality drop in game and a big performances boost, it's a good compromise.

I think that a benchmark like 3Dmark has to provide a no-optimisations mode, an IHV optimised mode and a fly mode to everyone. In the middle of the benchmark it has to provide everyone the possibility of ending the bench and flying in the scene. It's the only way to see if the IHV's optimisations are good or not. Example: procedural texturing. An IHV could use a simpler algorithm. This new algorithm could look good at the standard viewing point. But what happen when we zoom on the procedural texture ? With a fly mode, it's very easy to check this.

 

[Evildeus]

How about "rules" like this: Thou shalt not dishonestly represent thy products.

Yes, but then it depends on what is honest representation

1) It's possible to cheat benchmarks in ways you cannot cheat games

Disagree, depends on the game and bench. If the bench is randomly rendered, whereas the game is on trail (some race games for exemple) you can do more on games than on benchs. It's not a general statement.

2) Benchmarks are meant to compare and investigate hardware; games are meant to be played

Great, thus you never bench a game don't you? And to compare and investigate, doesn't mean that you shouldn't do "optimisation"? To bench is to have an idea on how your hardware will run on games (as far as i know) and then you it would compare on games (otherwhy it's not relevent to someone buying a game card), thus know how good it would be to play with this card over this other one.

3) Benchmarks are designed to show average-case performance for 3D software running under an API. The great majority of games are average-case, meaning there is no application-specific, performance optimization for them at all in 3D card drivers. Therefore, a benchmark which is is either cheated or optimized for will return performance values incongruent with the product's performance in the majority of 3D software available. This means it will return a *false* performance picture.

Well for this, the "average-case performance" doesn't exist, there's different path give some or or less optimised applications. And for a bench to be representative of "the majority of 3D software available" it should use the "average-case" of available softwares not being future proof isn't it?

The difference is as clear as the difference between a benchmark and a game.

Let's take a far fetched exemple
If a card is optimised for all current games but not for a specific benchmark, that this card has the best performance on games at equivalent IQ, but far less in the specific benchmark, what would you say? Optimising for the benchmark is cheating because it will "return performance values incongruent with the product's performance in the majority of 3D software available"?

PS: If the drivers are optimised for games, then to show the "average-case performance" it should be also on benchmark

 

[Althornin]

Let's take a far fetched exemple
If a card is optimised for all current games but not for a specific benchmark, that this card has the best performance on games at equivalent IQ, but far less in the specific benchmark, what would you say? Optimising for the benchmark is cheating because it will "return performance values incongruent with the product's performance in the majority of 3D software available"?

At least you recognize that it is far fetched.
However, you use this as an argument in your favor, while the non far fetched cases (ie, the common ones) are completely agaisnt you. Try again please.
You dont "win" by ignoring the many holes in your reasoning to poke a single worst case scenario through someone elses. you win by fixing the holes in your argument...which your post fails to do.

 

[K.I.L.E.R]

Does anyone know a good point about 3dmark 2003?

The way it purely tests your vidocard with little CPU overhead compared to the previous versions of 3dmark.

If I want to test how fast a certain architecture is against another architecture using a standard path or specific paths as long as the developer has implemented them I should be allowed to run the test knowing that the results I get have not been manipulated in any way by anyone other than myself.

Inserting pre-set clip planes and other "optimisations" will manipulate that test and I will get manipulated results which I never intended.

3dmark03 runs the way the developer intends it to run. For an IHV to sneak cheats into the drivers will manipulate the results from what I would normally get.

Hence, it's cheating.

 

[Bouncing Zabaglione Bros.]

Does anyone know a good point about 3dmark 2003?

The way it purely tests your vidocard with little CPU overhead compared to the previous versions of 3dmark.

I agree completely. 3DMark 2003 is designed to produce a metric of performance by placing different types of heavy load on the graphics card. This is signifcantly different from what games do.

For Nvidia to say this is irrelevent to them is akin to them driving (instead of running) around a marathon course, and then claiming that no one would actually *run* the 26 miles when we have cars. To make such a claim is to (deliberatly) not understand the point of such a test, or the context of it's results. To cheat on them is to lie about your performance, just as finishing the marathon in a car is a lie.

 

[K.I.L.E.R]

nVIDIA have been claiming superior video cards to the competition and when a benchmark that tests just that, nVIDIA fail to put the money where their mouth is.

Instead they revert to manipulation and trickery to make it look like they have superior hardware.

 

[ET]

Evildeus, perhaps you'd agree to the idea that an optimisation that helps only a specific visual path is a cheat? If a certain game demo is optimised, but that doesn't help actual gameplay in that game, then the optimisation is just meant to give people the wrong impression of the card (that is, that it's faster than it really is).

That said, I've pretty tired of this discussion of NVIDIA's cheating. Okay, they cheated, don't buy their products. Tell your friends, if you want. But discussing it over and over in a forum where everybody already knows about it and has their opinion is a waste of time, IMO.

 

[WaltC]

Yes, but then it depends on what is honest representation

Yes, sort of like claiming to have tried pot without inhaling it...

Honest: Your drivers approach the benchmark just as they approach the majority of 3D games on the market.

Dishonest: Benchmark is special cased (either though optimization or workload-reduction cheats) in your drivers and provides a performance picture incongruent with what your hardware does in the majority of shipping 3D games (sometimes any 3D games.)

1) It's possible to cheat benchmarks in ways you cannot cheat games

Disagree, depends on the game and bench. If the bench is randomly rendered, whereas the game is on trail (some race games for exemple) you can do more on games than on benchs. It's not a general statement.

How can you say "I disagree" and then say "it depends"... nVidia proved how it could cheat a benchmark in a way impossible with a 3D game (I get weary of talking about inserted clip planes, etc.) In a game those characteristics would be considered bugs and be fixed.

Great, thus you never bench a game don't you? And to compare and investigate, doesn't mean that you shouldn't do "optimisation"? To bench is to have an idea on how your hardware will run on games (as far as i know) and then you it would compare on games (otherwhy it's not relevent to someone buying a game card), thus know how good it would be to play with this card over this other one.

Still doesn't change the fact that you can't "play" benchmarks... The purpose of games is that they be played--regardless of what else you do with them. The purpose of benchmarks is to investigate and compare hardware--and you can't play them even if you want to. Heh... you can certainly play *with* them if you want to...

Well for this, the "average-case performance" doesn't exist, there's different path give some or or less optimised applications. And for a bench to be representative of "the majority of 3D software available" it should use the "average-case" of available softwares not being future proof isn't it?

Average-case = the way your hardware runs most 3D games--ie, without driver-related performance optimizations.

Special-case = the way your hardware runs a tiny handful of all of the 3D games available, with driver-related performance optimizations, vendor paths, etc.

There is no such thing as future proof.

Let's take a far fetched exemple
If a card is optimised for all current games but not for a specific benchmark, that this card has the best performance on games at equivalent IQ, but far less in the specific benchmark, what would you say? Optimising for the benchmark is cheating because it will "return performance values incongruent with the product's performance in the majority of 3D software available"?

Actually, as I've said before, if an IHV includes performance optimizations in its drivers for > than 50% of all shipping 3D titles, then optimizing in a similar fashion for a benchmark would present the average case.

However, no IHV's do this--even get close to this--and so it's a moot point. "Far fetched" as you say... The current state of IHV driver development is such that "average case" = no optimization for benchmarks (and of course certainly no cheats.)

 

[Himself]

Let's see, if a 10 year old is asked to make up his room and he just tosses everything under the bed, what would that be? Cheating or optimizing?
Never mind the rotting pizza crusts and dirty clothes, if you can't see the mess then everything is good right?

 

[martrox]

Let's see, if a 10 year old is asked to make up his room and he just tosses everything under the bed, what would that be? Cheating or optimizing?
Never mind the rotting pizza crusts and dirty clothes, if you can't see the mess then everything is good right?

But....after 2 weeks you wonm't need to see it....the smell will tell you it's there......

Hmmmm....maybe nVidia needs to add the feature "Smell-o-rama" to their FX line of cands!

 

[WaltC]

Let's see, if a 10 year old is asked to make up his room and he just tosses everything under the bed, what would that be? Cheating or optimizing?

I suppose it would depend on whether his parents look under the bed...