ExtremeTech Article Up

[jjayb]

Heh, Chalnoth still waiting for the drivers that improve the nv30 p.s. 2.0 scores? Are they here yet?

 

[rwolf]

Makes sense why they called the driver Detonator "FX". Their benchmarks use special effects to get better results.

 

[OpenGL guy]

I have the drivers(44.03) installed now and am getting the 'no clip' artifacts in the sky of SeriousSam(first one, not SE). I have a screenshot if anyone can host it.

Good job finding another "optimization"!

 

[lar2r]

Crap - I knew this was bigger than just 3dMark03.

Games didn't seem that much faster, even though websites were reporting improving performance on the FX cards.

 

[WaltC]

I have the drivers(44.03) installed now and am getting the 'no clip' artifacts in the sky of SeriousSam(first one, not SE). I have a screenshot if anyone can host it.

Good job finding another "optimization"!

Tip of the 'berg, huh? Oh, brother...next thing we'll discover is that this has been SOP for nVidia for years....

 

[MuFu]

They have been recording geometry and software culling for ages - the need just wasn't that great in the GF3/GF4 era (when they had it relatively easy) so there was little point in doing such things extensively considering the risk of getting caught.

Both companies use app-specific optimisations, we've just never seen a "smoking gun" scenario like this before.

MuFu.

 

[OpenGL guy]

I have the drivers(44.03) installed now and am getting the 'no clip' artifacts in the sky of SeriousSam(first one, not SE). I have a screenshot if anyone can host it.

Good job finding another "optimization"!

Tip of the 'berg, huh? Oh, brother...next thing we'll discover is that this has been SOP for nVidia for years....

I dunno, I was only kidding in my post But, it's easier for me to believe it's a driver issue in this case than in the 3D Mark 2003 case. I mean, skipping color clears only when it's safe to and clipping the sky only when it's safe to (in benchmark mode obviously) is a little too coincidental.

 

[Geeforcer]

A couple of questions (if they were already answered in this or other threads, I apologize).

1) Does this rendering anomaly manifest itself with all Nvidia cards capable of running 3dmark03 for a specific driver version, or only with FX series?

2) On the hardware effected, does this occur in every driver release, or only in the newer ones?

3) Finally, if this behavior is limited to certain driver version(s), what is the performance delta between the "normal" and no-clipping drivers?

 

[WaltC]

I dunno, I was only kidding in my post But, it's easier for me to believe it's a driver issue in this case than in the 3D Mark 2003 case. I mean, skipping color clears only when it's safe to and clipping the sky only when it's safe to (in benchmark mode obviously) is a little too coincidental.

OK, I'll go with you on the Serious Sam thing... And of course I agree on 3dMark 03 as well. I'd rather think any further similarities that might pop up are driver bugs, too. I hope so--too depressing otherwise...

 

[lar2r]

Finally, if this behavior is limited to certain driver version(s), what is the performance delta between the "normal" and no-clipping drivers?

Let's see

43.30 performance really sucked ~2800
43.45 performance improved but NVIDIA using partial precision (or half arsed colors)
43.51 and 44.03 performance still great but colors look fixed.

I'd bet it would be the last time since I had low performance that I saw what this card (5800Ultra) could really do in DX9.

 

[Reverend]

It is a big f**king deal, whether ET came up with this article or not, or whether forums heat it up. It challenges every preconceptions we may have had about reading and trusting reviews.

Well, right now, it only challenges 3DMark scores.
IF it was proved nVidia used similar cheats in other things, such as Serious Sam timedemos and Codecreatures, then it would become a fricking, damn huge deal.
Then I'd be really disgusted.

Sorry for bringing up the following analogy :

<analogy>
When 911 happened, US had a stance whereby it feared that it could not only happen again to American properties and lives but that it could happen to other countries. Those who agreed with this stance probably approved of the US' invasion of Iraq.

Should the US wait for another 911 or should it make its position about the possibilities clear? Should its "pre-emptive" strike be justified?
</analogy>

The point is that if NVIDIA is doing this with 3DMark03 and it has been proven to be so (I will not believe it's a driver bug as they claim), then it could do so with any benchmarks -- synthetic or games. If I record a Splinter Cell demo for the purpose of benchmarking this demo in video card reviews, NVIDIA can study my recorded demo and do the same thing.

What does this mean to a reviewer? That they are now "forced" to forever :

1) Run a game benchmark using a recorded gameplay demo
2) After running the game benchmark, play the game at the same scene as the recorded demo and check if the benchmark results translate into real gameplay results, with the reviewer varying his POV both slightly as well as wildly, to check for inconsistencies

Summary : Doubt has been introduced by NVIDIA wrt 3DMark03. I don't know about the rest of the reviewers out there but this doubt will mean I will need to do the above for every one of reviews from now on. Sure, it means more work for me but you have to wonder why this extra work should/would be necessary.

The "huge deal" isn't about whether NVIDIA will attempt the same with recorded game demos but that they have simply introduced such a possibility. Can you tell me honestly that you won't have ANY measure of doubt from now on when you see benchmarks of Games X, Y and Z that uses recorded gameplay demos? Do you see the implications, the far reaching consequences, the whole concept of video card benchmarking and reviewing?

If they could apply such a thing to real games too and get similar performance boosts, then I'd be delighted. But it is 100% obvious those cheats are specifically for static paths, so it's obviously impossible.

Again, do you read reviews and make purchasing decisions based on these reviews? Reviews = using static paths. Will you continue to trust reviews then?

However, considering it only influences 3DMark and some major websites didn't even use it during their NV35 review, I wouldn't make it that much of an issue.

NVIDIA can't have known "major" websites wouldn't be using 3DMark03 in their reviews of NV35. What is a fact (and it is a known fact to NVIDIA) is the kind of influence 3DMark03 has on the mindset of a lot of people.

That is, as I said, unless it was applicable to other timedemos too - then it'd become truly disgusting.

And how are you to know if this applies to other timedemos too, whether current timedemos or future ones? You will buy every video card and test them yourself to find out? Or you will read reviews? Will you trust such reviews without even wondering if the reviewer has now been forced to do extra work to verify that timedemo results = actual gameplay results? What if a reviewer do not mention "We can verify that the timedemo results translates into actual gameplay results"? Which reviewer do you trust? How many reviewers are there in this video card reviewing industry?

It is not so much about the fact that the only evidence now is 3DMark03 but the implications and doubt introduced. I repeat what I said -- you don't appear to grasp the gravity of the situation, about the bad possibilities that it introduces, about how reviews should be conducted now.

[edit]I should add that this doesn't necessarily mean that I will only have to be careful when reviewing NVIDIA hardware from now on. I will extend NVIDIA the "courtesy" of being pessimistic and that this can apply to any and every IHV and I will be careful with all hardware reviews I undertake from now on, be it NVIDIA, ATi or others. That is why it is so worrying to have this doubt introduced.

 

[stevem]

I think we should just ditch 3DMark as a benchmark.

Precisely what Nvidia wants, no...? The issue with 3DM03 is that Futuremark devs allow for test modes, revealling some tricky profiling Nvidia driver devs are able to come up with. No wonder they didn't like the direction of 3DM03 development. By exposing 3DM03 weaknesses to cheats, they expose their own driver code... The clincher is the progression of 3DM03 quirkiness with each subsequent driver revision. It's a systematic process. I could, again, recount the dismay of 3dfx driver devs when exposed to the Nvidia code base... All IHVs profie popular benchmarks. When this coincides with actual apps/game-play, the results are worthwhile. Some have a knack for playing with grey areas...

 

[OpenGL guy]

1) Run a game benchmark using a recorded gameplay demo
2) After running the game benchmark, play the game at the same scene as the recorded demo and check if the benchmark results translate into real gameplay results, with the reviewer varying his POV both slightly as well as wildly, to check for inconsistencies

You can simplify this by just creating custom demos for each game you want to benchmark. It would make results between sites hard to compare though.

 

[Ichneumon]

1) Run a game benchmark using a recorded gameplay demo
2) After running the game benchmark, play the game at the same scene as the recorded demo and check if the benchmark results translate into real gameplay results, with the reviewer varying his POV both slightly as well as wildly, to check for inconsistencies

You can simplify this by just creating custom demos for each game you want to benchmark. It would make results between sites hard to compare though.

I had just been about to post the same thing myself.

While it makes result comparisons difficult, reputable reviewers/sites can always have their own set of demos for the games they bench, which they do Not make publicly available.

Heck, they can have a one or more they don't make available, and a one or more they do or whatever... then anything like this would become rather obvious as any performance anomalies due to "optimizations" with the publically available demos would be readily apparant against the in-house demos.

While I definately understand Rev's take on things, it seems a little on the extreme side of looking at it... reviewers just need to have their own set of demos that they know well which they use in conjunction with other more readily available demos so they can keep their finger on performance gains vs. anomalies.

 

[Dave Baumann]

2) On the hardware effected, does this occur in every driver release, or only in the newer ones?

There are differences between drivers revisions. The 'clipping'* occurs in GT2 under 43.45 drivers, but not in GT4, move to 43.45 drivers and it occurs in both GT2 and GT4. Not tested 44.03 yet and so far I've looked at this on an NV31.

(*) Disclaimer - I'm not suggesting that this is actually clipping, but this describes the effect that I'm referring to.

 

[Colourless]

There 'could' be a significantly larger number of driver 'bugs' affecting Proxycon and Trolls Lair. There is a reasonably probiblity that there might be stencil volumes that could be culled without negetive inpact to the scene (i.e. the volume doesn't hit a surface on the screen). If the volume is close to the camera, culling the volume can save a substanaial amount of fillrate.

 

[Pete]

OK, it's obvious that some deception is occurring here. The question is, who (directly) is to blame? I'm hearing lots of of "Nvidia is cheating", "Nvidia is deliberately deceiving us". And of course the usual suspects are holding this incident up as proof that Nvidia as a whole is somehow ethically bankrupt.

How many employees in the Nvidia driver-writing group? How many levels in the organization below CEO are the driver writers? How large is the driver-writing group in relation to the whole company? Think Jen-Hsun Huang is directing driver writers in their tasks? I'd bet that he has other things to worry about, like running a Fortune 500 company.

Is it possible that a low-level lead engineer in the driver writing group directed his team to get the benchmark scores up, and it became a minor side-project of a lone driver writer? Is it possible that noone else was aware of the cheats he implemented?

Every past attempt to cheat at benchmarks has been uncovered and has blown up into a media fiasco. Surely the decision-makers at Nvidia know this and are not dumb enough to encourage this sort of behavior. The idea that driver cheating is a some sort of "corporate policy" endorsed by Nvidia is absurd (or Nvidia leadership is just fantastically stupid).

Again, it's clear that the Nvidia drivers have been manipulated to unfairly optimize for a benchmark. I'm just saying that there's no need to prosecute an entire organization based on what could be the actions of a very few individuals.

I seem to recall a few nVidia employees not of the driver-writing team who claimed the FX could output eight pixels per clock at launch.

I have a hard time objecting to any supposed cheating in 3DMark.

3DMark is a useless benchmark to start. "Cheating" can't make it any less useful.

The way I see it, whether the benchmark shows the GeForce FX in far too negative a light (by using shaders that the FX is particularly bad at), or the benchmark shows the FX in too positive a light, it's just as invalid.

That's some extremely short-sighted logic, Chalnoth. It's useful just because most sites use it as a comparative benchmarking tool. COMPARATIVE, not absolute. And its results seem to coincide with other benchmarks, like xbit's RightMark, so it's not exactly alone in the insights its results offer.

BTW, I believe the FX was built around shaders--CineFX, no? 128-bit color? Don't tell me just because the FX is slow at what it's advertised to do, that we can't hold it to account. If you believe the benchmark to be invalid, then tell review sites, don't cheat.

For heaven's sakes, someone host Ben's picture! Ben, did you try PM'ing Wavey?

 

[Mariner]

I have a hard time objecting to any supposed cheating in 3DMark.

3DMark is a useless benchmark to start. "Cheating" can't make it any less useful.

The way I see it, whether the benchmark shows the GeForce FX in far too negative a light (by using shaders that the FX is particularly bad at), or the benchmark shows the FX in too positive a light, it's just as invalid.

What they are cheating in is irrelevant. It is the cheating that counts.

If NVidia are willing to deliberately mislead/lie to the consumer about the performance of their hardware in a piece of software (which can be purchased), then they could conceivably be willing to lie about anything else.

If you accept cheating in any piece of Benchmarking software, you are also accepting cheating in Quake 3, Doom 3, Half-Life 2 etc. After all, if you know someone is a proven and deliberate liar, why should you believe anything they say?

 

[Patric Ojala]

When is Aquamark 3 coming out so we can just forget about 3DMark03?

The screenshots of Aquamark 3 look great and I hope the benchmark in general is as polished and thorough. The thing in the Aquamark 3 articles I was a bit negatively surprised with was the somehow 'good thing' that Aquamark 3 does different rendering depending on what hardware is running it. This would mean that Aquamark 3 benchmarks well the performance of a game that uses the Aquamark 3 technology (engine, artwork etc.), but not "apples to apples" hardware performance. In that case there will still be great need for 3DMark03, even if Aquamark 3 joins the range of available benchmarks.

I would therefore strongly recommend the implementation of at least a non-default benchmarking option in Aquamark 3, that forces the same rendering, no matter what hardware is running it. Alternatively they could copy our idea, and use fall-back shaders that produce the exact same rendering, but uses multiple rendering passes producing it on hardware with lower shader version support.

This topic might have been discussed here already, but if not, this is something to consider.

 

[jpaana]

Also, I insist on my POV that 3DMark "sky before everything" is not acceptable.

It's very common technique for drawing something that's supposed to be far away (so that everything is drawn over it) without sacrificing Z-range (sky cube isn't very friendly for Z as rotation changes the Z-range used pretty much). So first sky without Z test and then everything over it normally.