ExtremeTech Article Up

[tamattack]

Also be interesting to see if this affects older cards (eg: GF4) using DetFX.

 

[Reverend]

Dave's post on this plus my short comments. Just as a matter of info.

B3D had known about this for about a week now and Dave, while he wanted to make an article of this similar in nature to ExtremeTechs, felt that it was best that ET, being a Ziff-Davis company, would be better equipped to handle the uninevitable, uh, backlash and consequences.

 

[Lezmaka]

The problem is primarily with GT4 which the GF4 can't even run. Since GF4 can run the other tests which have some "anomalies", who knows. Maybe someone can test that for us?

Seems like reviewers might have to resort to making their own timedemos for each review to make sure nothing fishy is going on.

 

[Seiko]

More publicity required!

Dave's post on this plus my short comments. Just as a matter of info.

B3D had known about this for about a week now and Dave, while he wanted to make an article of this similar in nature to ExtremeTechs, felt that it was best that ET, being a Ziff-Davis company, would be better equipped to handle the uninevitable, uh, backlash and consequences.

Well I just hope this discovery gets plenty of publicity. I have feeling if only ExtremeTech publish or make a comment nvidia may just ride out the storm as they've already got 99% of the web sites showing them winning the 3Dmark benchies! Of course I won't hold my breath for Toms or H's article

 

[Evildeus]

Little OT:

There's one more thing that i do not really like. I was looking at the beta program guidelines and found this:

The BETA Program is an opportunity for joint research between Futuremark and BETA members to exchange information and ideas about the future outlook of the computer industry.

The key policy of the BETA Program is neutrality and participation is open to all applicable companies. Cooperation is always carried out with utmost confidentiality.

Now i look back at this affair and i don't see anywhere any information from Futuremark on this issue. Why?

Is that to say that Extremetech/B3D will be the official spokeman for Futuremark when this kind of issues get out? Can Extremetech/B3D can do the same thing onther beta members products given the confidentiality requirements?

I think that there's a conflict of interest over here. Hope to see some clarifications from you guys

 

[Seiko]

Little OT:

There's one more thing that i do not really like. I was looking at the beta program guidelines and found this:

The BETA Program is an opportunity for joint research between Futuremark and BETA members to exchange information and ideas about the future outlook of the computer industry.

The key policy of the BETA Program is neutrality and participation is open to all applicable companies. Cooperation is always carried out with utmost confidentiality.

Now i look back at this affair and i don't see anywhere any information from Futuremark on this issue. Why?

Is that to say that Extremetech/B3D will be the official spokeman for Futuremark when this kind of issues get out? Can Extremetech/B3D can do the same thing onther beta members products given the confidentiality requirements?

I think that there's a conflict of interest over here. Hope to see some clarifications from you guys

Although your primary point about conflict of interest certainly raises questions I think the OT statement isn't really justified. With crud like this
http://www.theinquirer.net/?article=9486 appearing on websites these kind of perceived improvements will do wonders for sales! Although I'd certainly brand it as unethical it does make perfect business sense as mass sales are simply derived from king of the “benchiesâ€￾!

Do I expect nvidia to change? in all honesty no. Do I think ATI are really that much different? I doubt that too! The whole optimisation area open up a huge can of worms as personally I see nothing wrong with optimisations for popular game engines. Personally I feel Nvidia have crossed an already extremely grey line anyway as they fail to render the scene correctly at all angles and at all times but then look at the optimisations ATI and Nvidia use with their new AF algorithms. Again optimising is a grey area and although in this case Nvidia in my opinion has gone too far it does help sell units so don’t expect it to stop anytime soon!

 

[LeStoffer]

Well, I liked nVidia's initial response:

nVidia believes that the GeForceFX 5900 Ultra is trying to do intelligent culling and clipping to reduce its rendering workload, but that the code may be performing some incorrect operations.

I knew that the NV35 is a great piece of hardware, but I didn't know that it is so darn awesome that it can intelligent - as a living, thinking entity! - make its own decisions and thus even defy its creator, nVidia.

 

[Evildeus]

seiko,

Sure that doesn't mean that Nvidia is not at fault. But i think it something we should have in mind and discuss, because it's not harmless.

 

[demalion]

Okay, so let me get this clear...

That's ALL there is to it?
There better be more, otherwise the hype about those cheats is vastly unjustified

Vastly unjustified?! Uttar, the concern is not the cheat method being excessively complicated, but that it achieves an unrepresentative comparison.

This is a very disappointing post on several levels. Not because of your cheating analysis, but the proposition you start off with that you are attempting to support with it.

Please consider a quote from the article:

If this were a "3D Guided Tour" and the goal was to make the scene render as quickly as possible, with no regard to any real-world performance correlation, then this type of optimization would be fine. But that's not the goal of 3DMark. It's a synthetic benchmark that measures performance, designed to indicate how well a 3D GPU can render DX8/DX9 game code. In a game where you have six degrees of free movement, with a user controlled (vs. fixed) camera path, static clip planes would not work at all.

This is not a general case optimization, and seems to have been progressively worsening concurrent with performance increases. The trend and indication has been established already, and the improper representation has already been done. The "hype" is not "unjustified"...nVidia is in the position of having clearly shown that their optimizations have been taking inappropriate shortcuts, in a clearly dedicated way that is uniquely dependent on "rail" demo playback. That much seems clearly established. Tell me, how do you accidentally optimize in such a way to benefit a gamer's experience? The goal of the optimization matters, Uttar, and this set of optimizations is for benchmark results that influence buying decisions but do not represent what the user will get in actual game play.

What is not being established is the actual general case performance of their cards in 3dmark 03, and this is discussed exactly in the article as well, by how the 3dmark 03 performance results do not represent general case, even other shader limited ones. The problem here is that this misrepresentation is achieved intentionally by nVidia, and is not the fault of 3dmark 03 except inasmuch as not being completely cheat proof.

What is established, and what is not, is the problem, because nVidia proposed the first as the second. That seems to be a rather blatant deception.

Let's face it, the sky problem WILL be gone in the next driver revision. And I mean, really gone. It's so darn fricking easy to fix it isn't even funny anymore. As I explained in another post, it should be possible to *cache* a DIP call and draw it after everything else. And then, it's perfectly undetectable.

The problem is the special case nature would still make it unacceptable. Prior view on the topic.

However, it is conceivably possible to cache and evaluate draw order as a general case optimization, but the problem here is that we don't have the performance results for just that being used to represent the product, we have what is already established to be done instead. It is your dependence on "they can do something different" and "only this many % performance increase, I'd guess" to support that this doesn't matter that is uniquely disappointing.

The buffer clearing problem surprises me most however. From the looks of the screenshot, it seems like they're still clearing Z, but not color. Odd, considering the NV3x got Fast Color Clear. So yes, they're cheating, but I'd be surprised if they got more than a 3% boost from that.

OK, but your surprise wouldn't change that they're cheating.

Oh, sure, there may always be other things, but I guess I was right. I'm not particularly shocked by all this. The performance boost they're getting is probably lower than 10%, which would indicate they might still be able to be on par with the Radeon 9800.

The problem is having to guess by how much they misrepresented, Uttar.

While it does proof nVidia is cheating, and it would indeed be better if they stopped, this ain't THAT much when you think about it.

I don't think that is an objectively valid statement.

And heck, anyone drawing the sky before everything shouldn't expect GPU companies not to cheat IMO. It's just too unoptimal.

That's not the only cheat, Uttar, and if it can successfully be done in the general case, or is openly presented as an option (discussed in the article, as well...a balanced piece, IMO EDIT: hmm...except that the Quack mention, while necessary, was significantly incomplete), it would indeed be more valid.

But that's "coulda, woulda, shoulda, if, but, maybe", Uttar, and still leaves the rest of what they did.

Hasty example:

You purchase a tool that says it offers a certain specification of...let's say...torque and rpm.

You come to find out that you can't use it well for tasks that should be within the capabilities of its rated spec.

The reason for this is because the manufacturer used a motor that offered the rated spec for specific circumstances only, ones that are not realistic or representative of any common usage.

Do you blame yourself for not having the tools to immediately find out that the rated spec was not correct?
The store for allowing the tool to have that spec printed on the box?
The testing lab for not having enough power to dictate that everyone must adhere to their standards for specifications?
The manufacturer for circumventing the standards in ways that other manufacturers do not, in order to state a specification that is inflated?

This manufacturer has dedicated considerable resources to having people do the third.
You seem to propose anything but the 4th is suitable, and in fact the 4th is "vastly unjustified", even in view of what this manufacturer is dedicating resources to convince people to do.
Why is that?

EDIT: Fixed link.

 

[AJ]

When is Aquamark 3 coming out so we can just forget about 3DMark03?

*smile* Solomon, I'm afraid it won't be this simple. The actual issue is that no matter which performance metrics are used the same elements will have to be considered. I.e. when Aquamark 3 comes out the Massive guys will get to deal with the same issues.

Cheers,

AJ

 

[Mark]

When is Aquamark 3 coming out so we can just forget about 3DMark03?

*smile* Solomon, I'm afraid it won't be this simple. The actual issue is that no matter which performance metrics are used the same elements will have to be considered. I.e. when Aquamark 3 comes out the Massive guys will get to deal with the same issues.

Cheers,

AJ

Yes, I agree. This goes far beyond 3DMark in my opinion.

 

[AJ]

Now i look back at this affair and i don't see anywhere any information from Futuremark on this issue. Why?

We'll write a more official response to this shortly. The ExtremeTech article went live between 2-3 am local time so it will take a while for us to react. In general we'll be happy to explain or elaborate on any technical questions that will be presented to us, but typically we seldom take the soap box ourselves (as were software company, not a media house).

Cheers,

AJ

 

[jjayb]

Hmmn, wonder what kind of optimizations are in code creatures benchmark.

****off topic******
You know it just occured to me that I've never heard anyone complain about sites benching code creatures in the reviews. Only 3dmark03. Maybe because Nvidia hasn't downplayed it yet? I'm amazed at how many people follow Nvidia's every wish. Do they forget that they are the customer and Nvidia is the company?
****end of my off topic pondering******

 

[Evildeus]

Thx AJ for the responses. More Qs 8)

Are you saying that one of the reason of web site in the beta members is to "take the soap box" by themselves?

Didn't you (Futuremark) did the job (ie press release) when it was the first issues on 3DMark03 with the GF FX?

For my information, can a beta member (ie the media houses) do the same thing on any beta member? Or are they bond by confidentiality?

Now i look back at this affair and i don't see anywhere any information from Futuremark on this issue. Why?

We'll write a more official response to this shortly. The ExtremeTech article went live between 2-3 am local time so it will take a while for us to react. In general we'll be happy to explain or elaborate on any technical questions that will be presented to us, but typically we seldom take the soap box ourselves (as were software company, not a media house).

Cheers,

AJ

 

[Doomtrooper]

When is Aquamark 3 coming out so we can just forget about 3DMark03?

So doing similar things to Aquamark makes that a better 'benchmark'

If they need to hack a Benchmark to get frames, they need to hack game titles to get frames...the hardware doesn't magically fix itself when a game is ran.

 

[micron]

When is Aquamark 3 coming out so we can just forget about 3DMark03?

So doing similar things to Aquamark makes that a better 'benchmark'

If they need to hack a Benchmark to get frames, they need to hack game titles to get frames...the hardware doesn't magically fix itself when a game is ran.

True Doomtrooper...why is that so hard for people to understand?

 

[BoardBonobo]

Well it's certainly bad for nvidia but how damaging is it for FutureMark? It kind of invalidates their current product and throws doubt on the validity of all the acores in their database.

Two generations of 3DMark have shown themselves to be very susceptible to 'hacks' that can greatly skew the results, can any of them be trusted anymore.

Though this a bad thing for nvidia, it is a good thing in the long run as it highlights the fact that our reliance on synthetic benchmarks, as gauge to performance, is severly misplaced in ill conceived.

Synthetic benchmarks are just a way for a driver team to show how great they are at optimising for a specific piece of code and how cunning they are at hiding the fact. And that applies to all gpu makers.

 

[Andergum]

There needs to be a "Underwriters Laboratories" type company for ensuring benchmark integrity I guess.

 

[Richthofen]

When is Aquamark 3 coming out so we can just forget about 3DMark03?

So doing similar things to Aquamark makes that a better 'benchmark'

If they need to hack a Benchmark to get frames, they need to hack game titles to get frames...the hardware doesn't magically fix itself when a game is ran.

why do they? What has 3DMark to do with typicall games.
3dMark is more kind of PS1.4 test than anything else. Guess which company colaborated with them to do this?

I don't care if Nvidia cheats in 3DMark at all. I know they do it and i know others will do the same as long as these stupid synthetic benchmarks exist. Time do drop that crap soon.
Same with CPUs. I am sick of reading reviews with over 50% of the tests done with benchmarks like sysmark, pcmark, sandra and so on.

 

[micron]

Big companies like Dell will purchase video cards for their machines based on the premise of things like 3DMark scores...Nvidia knows how important synthetic benchmarking is to their livelyhood....And to the guy who posted above me...I like 3DMark, it might not mean anything to you, but it does to me, and alot of other people....you must have gotten a really crappy score or something