Mr. Ojala

[K.I.L.E.R]

WOAH!

I just read the front page and now you guys no longer recognise what nVIDIA have done as a cheat but as an optimisation.

IMO it is still a cheat.

What's up with the change of heart?

Is there something going on that we don't know about?

Please tell me that I just misunderstood the entire news post.

Mr. Carmack and Mr. Sweeney both recognise what NV had done as a cheat, not an optimisation.

I am seriously confused now. :?

 

[epicstruggle]

I think we will only be able to speculate what really happened. Im sure the workers of fm have been "advised" as to what they can/can't say. Im also suprised that they reversed themselves. I would have rather they not used the word cheat from the beginning. Anyways, just my 2 cents.

later,

 

[Reverend]

Is there something going on that we don't know about?

Surely this should be obvious.

 

[K.I.L.E.R]

Is there something going on that we don't know about?

Surely this should be obvious.

Assuming anything regardless of how obvious it is can be foolish in most situations especially if you don't know the full story.

Better ask before I drop a conclusion like many have done. It's already been answered on Mr Ojala's forum.

By the looks of it there is nothing that is going on behind our backs other than FM appeasing NV (to stop the fighting\arguments\whatever you want to call it) which I am sure many have come to the conclusion before Mr Ojala had said anything.

 

[Reverend]

http://www.beyond3d.com/forum/viewtopic.php?t=6230

Q: Does this mean that in the future you will not make patches for 3DMark03 (or 3DMark2001) in order to reveal cheating?
A: We might release further patches to 3DMark03, if a need for preventing driver optimizations appear in the future.

The Q asks about "cheating" but the answer talks about optimizations. I mean, c'mon Patric, stop being so damn diplomatic and politically correct!

 

[K.I.L.E.R]

http://www.beyond3d.com/forum/viewtopic.php?t=6230

Q: Does this mean that in the future you will not make patches for 3DMark03 (or 3DMark2001) in order to reveal cheating?
A: We might release further patches to 3DMark03, if a need for preventing driver optimizations appear in the future.

The Q asks about "cheating" but the answer talks about optimizations. I mean, c'mon Patric, stop being so damn diplomatic and politically correct!

Thanks for providing me the link.

 

[CorwinB]

Futuremark Statement

For the first time in 6 months, FutureMark and NVIDIA have had detailed discussions on how to make the NV3x line of GPUs perform better than it should.

After being bullied and strong-armed, Futuremark now has a deeper understanding of its interest, which is to let NVIDIA get away with cheating. In the light of this, Futuremark redefines "cheat" to "application specific optimization", but only as far as NVIDIA's drivers are concerned.

Futuremark's perception of the world of 3D Graphics have changed dramatically with the latest cash incentive from NVIDIA. Just like for CPUs, each GPU maker has a different amount of cash available to buy independant benchmark makers. For example, if AMD gave us tons of cash, we would modify Futuremark's PCMark2002 to have Athlons run faster on it.

3DMark03 was designed as as un-optimized DirectX test because we thought that was the only way to provide meaningful performance comparisons. Because NVIDIA couldn't produce quality hardware to save their life and cheated in our product instead, we felt obliged to update it instead of welcoming the cheating. Silly us !

However, after threatening us with a lawsuit and offering us tons of cash to bend over and screw our end-users, NVIDIA suggested us that a different approach to game performance benchmarking might be needed, where NVIDIA-specific code path optimizations and clip-planes could be included directly in the code source, as opposed to having to be put in the Cheatonator drivers.

NVIDIA Statement

NVIDIA pays many developers to optimize games for GeForceFX, because its architecture can't compete without low-quality shaders and crap IQ. These optimizations (including lowering shader quality, arbitrary clipping plane and no Z-clearing) are the result of bribery and outright cheating and deception. This is the approach NVIDIA would have preferred also for 3DMark03.

Joint NVIDIA-Futuremark Statement

Both NVIDIA and Futuremark want to define 2 set of clear rules about benchmarks : one for NVIDIA, and one for the rest of the industry. We believe that those rules will prevent the kind of unfortunate situations where an NVIDIA crappy product is accurately described as such and will allow NVIDIA to further lie its way to the top.

 

[Eolirin]

Corwin, lol!

 

[Striker]

Futuremark Statement

For the first time in 6 months, FutureMark and NVIDIA have had detailed discussions on how to make the NV3x line of GPUs perform better than it should.

After being bullied and strong-armed, Futuremark now has a deeper understanding of its interest, which is to let NVIDIA get away with cheating. In the light of this, Futuremark redefines "cheat" to "application specific optimization", but only as far as NVIDIA's drivers are concerned.

Futuremark's perception of the world of 3D Graphics have changed dramatically with the latest cash incentive from NVIDIA. Just like for CPUs, each GPU maker has a different amount of cash available to buy independant benchmark makers. For example, if AMD gave us tons of cash, we would modify Futuremark's PCMark2002 to have Athlons run faster on it.

3DMark03 was designed as as un-optimized DirectX test because we thought that was the only way to provide meaningful performance comparisons. Because NVIDIA couldn't produce quality hardware to save their life and cheated in our product instead, we felt obliged to update it instead of welcoming the cheating. Silly us !

However, after threatening us with a lawsuit and offering us tons of cash to bend over and screw our end-users, NVIDIA suggested us that a different approach to game performance benchmarking might be needed, where NVIDIA-specific code path optimizations and clip-planes could be included directly in the code source, as opposed to having to be put in the Cheatonator drivers.

NVIDIA Statement

NVIDIA pays many developers to optimize games for GeForceFX, because its architecture can't compete without low-quality shaders and crap IQ. These optimizations (including lowering shader quality, arbitrary clipping plane and no Z-clearing) are the result of bribery and outright cheating and deception. This is the approach NVIDIA would have preferred also for 3DMark03.

Joint NVIDIA-Futuremark Statement

Both NVIDIA and Futuremark want to define 2 set of clear rules about benchmarks : one for NVIDIA, and one for the rest of the industry. We believe that those rules will prevent the kind of unfortunate situations where an NVIDIA crappy product is accurately described as such and will allow NVIDIA to further lie its way to the top.

I have uploaded the above test to a personal page, with the original statement to another, and start sending people there to "compare" the two pages...Thanks for the nice work, mate =)

 

[CorwinB]

I have uploaded the above test to a personal page, with the original statement to another, and start sending people there to "compare" the two pages...Thanks for the nice work, mate =)

You're welcome. Could I have the URL to your page ? Would save me some Copy/Paste work...

 

[Striker]

Ofcourse!

For reference purposes only, ofcourse (I'll add it to my sig in some mins )

Original statement:

http://www.modszone.com/various/otenet/docs/FM_NV_statement.html

Your statement:

http://www.modszone.com/various/otenet/docs/FFM_NV_statement.html


Use the URL freely, the server is always up, and it ll help you somewhat

 

[antlers]

This ended up being a very serious issue for NVidia, since if they engaged in actual "cheating" to create an incorrect impression of the performance of the NV35 at launch, it opens them up to shareholder lawsuits if the product later proves a failure (I think that such shareholder lawsuits are much more of a concern to them then any possible consumer lawsuits, especially since they do not sell directly to consumers, and also more important than any marketing harm done by the actual test results). In the face of the possibility of shareholder lawsuits, it became very important to NVidia to have Futuremark retract the accusation of "cheating", which surely would have been used by plaintiff's in any future shareholder lawsuit. The text of the statement makes it clear that "application-specific optimizations" means cheating as far as Futuremark is concerned, while removing the connotation of fraud that was so potentially damaging to NVidia.

 

[kyleb]

and here i was hopeing that nvidia would have to suck it up though such lawsuits and learn not to cheat in the future. silly me. :?

 

[demalion]

OK, I mentioned this in the thread about the Press Release. I'll refocus it as a suggestion to FM, by dropping the beginning and proposing it as someone hypothetically asking my opinion on the Futuremark statement and what Futuremark should do:

• Does this change what nVidia did from being a cheat?
  
  No.
  
• Doesn't this make Futuremark's statement unacceptable?
  
  No, I don't think this is the part that makes Futuremark's statement unacceptable. Hold on a sec before you explode because of that, though.
  
• (Trying not to explode)How could their statement possibly be acceptable!?
  
  If they were: 1) only talking about cards not using FX12, but using FP16 (the NV35 AFAIK) , 2) only talking about the shader processing that provides comparable output with FP16, 3) keeping shader discussion separate from clip plane issues.
  
• So, you're saying their statement is acceptable?
  
  No, they only even began to cover 3, but completely glossed over the crucial issue of clipping planes in legalese indirectness.
  
  Here, is where they did that:
  
  

  ...
  Because all modifications that change the workload in 3DMark03 are forbidden, we were obliged to update the product to eliminate the effect of optimizations identified in different drivers so that 3DMark03 continued to produce comparable results.
  ...

  
  Also, they did not even touch on 1) and 2), and ended up supporting the idea that they didn't matter. Completely unacceptable, even if unavoidable.
  
• So are you happy with Futuremark, or not!?
  
  Completely unhappy, disappointed, let down, and lacking confidence in their future benchmark applications, or the evolution of 3dmark 03 after some subsequent driver releases come out, because there are no assurances evident that I should feel otherwise.
  Specifically, this statement removes my confidence that 3dmark 03 will continue to be useful:
  

  ...
  However, recent developments in the graphics industry and game development suggest that a different approach for game performance benchmarking might be needed, where manufacturer-specific code path optimization is directly in the code source. Futuremark will consider whether this approach is needed in its future benchmarks.

  
  If there was condemnation of clipping planes AND all FX12 using cards using FX12 when not appropriate, this wouldn't have been unacceptable, it would have made 3dmark 03, IMO, an inferior benchmark.
  
  Since fp16, when the partial precision hint is applied, is part of the DX specification, supporting it would not be an invalidation of 3dmark as a DX benchmark. Here is an explanation of why I think so.
  
  
  But, the problem is that there isn't any condemnation, just capitulation, and that's what I'm viewing as the source of the problem since fp16 isn't the only issue involved:
  
  It's a big slap in the face for any company not doing the things nVidia was, because disregarding rules + non game representative shortcuts + applying legal pressure = successful way to cause Futuremark's products to be adapted to your liking. The problem is that this makes not cheating a bad business decision, and it is part of the responsibility of a benchmark company trying to be objective to achieve the opposite. Cheating is cheaper than good hardware design and general case software development, and what is the future direction of ATI and other IHVs going to be wrt to clipping planes and other "rail optimizations"?
  
• I'm confused...what's your final take?
  
  I think FM's statements here and current lack of action after it has the "nice" little side benefit of significantly affecting the future of the 3D industry, because they are the leading industry benchmark. Their benchmark software itself is still useful, at the moment, but the holes in their statements so far don't provide any assurance at all that the company as a whole will continue to be. It doesn't look like they're just including the "devrel" component I mention, it looks like they, and nVidia together, are including a "legal pressure component" as well . That doesn't belong in a technology benchmark under any circumstance.
  
  Leaves me disagreeing with a lot of what people are attacking them for (_pp IS part of the DX spec), but agreeing wholeheartedly that right now Futuremark is completely failing part of their customer base: the part that doesn't have a vested interest in nVidia's good name regardless of their actions.
  That would happen to include consumers who value their wallets more than nVidia devotion, and all nVidia's competitors. It doesn't necessarily include OEM customers of nVidia, because their wallets might be affected by nVidia's good name, and I think that's part of the reason FM made this statement.

Well, summarize what you think they should do, then

Here we are: representation-by-cheating-and-applying-legal-and-economic-pressure is a viable alternative to technological innovation at the moment. We have indication that FM wants things to be otherwise, but what is suddenly missing is assurance that they are able to deliver on that desire.
If legal pressure is controlling the statements allowed in public, and it seems clear that it is, then this is the time for them to take actions instead of making statements:

• Go the extra mile in opening avenues of communication with the user base that is betrayed by what is lacking with what has been stated.
• Provide assurances by genuinely focusing on delivering features that expose and highlight quality differences (with allowing lower quality, this is a responsibility you've directly taken on, and to a more significant degree, to remain an objective benchmark).
• Quickly and immediately make statements and deliver on actions, separate from the topic where there might be legal restrictions, that establish that the commitment to the above is real, and demonstrate that the effort in this regard is serious.

All of this is necessary because 3dmark 03 is indeed the industry leading benchmark...this is the responsibility of that position. Legal pressure might change what you can say, but it doesn't change what you have to deliver to be worthwhile in that position...if it does change what you deliver, your being in that position is not worthwhile,regardless of the reason.
The stance Futuremark took up until now delivered on that responsibility. Fine, Futuremark doesn't have the funding to maintain that stance in public statements, but the responsibility is still there, and there are other avenues open to fulfill it.

 

[g__day]

Apart form the obivous upset at NVidia tactics, does this mean FutureMark will allow NVidia to lower DX9 precision - as low as it wants 1) whenever it wants or 2) whenever image quality can be proven not to suffer?

 

[Nagorak]

It doesn't matter. If ATI couldn't gain marketshare during the 8 months their hardware was on top, they're going to be filing for bankruptcy in a couple years anyway.

 

[FMammal]

It doesn't matter. If ATI couldn't gain marketshare during the 8 months their hardware was on top, they're going to be filing for bankruptcy in a couple years anyway.

huh?

They're making a profit, which is more than about 70% of high tech companies nowadays can say. They're also doing it by selling better performing parts at lower prices, which leads me to believe they're doing it with smaller margins.

If Matrox, PowerVR, et al. can still stay in the game without any highly performing desktop parts at all I think ATi can handle themselves...

FM, on the other hand, is in dire straits, at least with those smart enough to grasp the situation over the past few weeks.

 

[DethWraith]

Well, all I know is, if I ever do any video card reviews, I'll use something other than 3dMark to compare cards. It may be considered the industry standard but I'm sure there are other ways to get non-biased benchmark results.

 

[Nagorak]

They're making a profit, which is more than about 70% of high tech companies nowadays can say. They're also doing it by selling better performing parts at lower prices, which leads me to believe they're doing it with smaller margins.

If they lose marketshare when they produce the best parts, what's going to happen now that they're back to being second best? I think people are just too stupid to buy anything but on brand name. I mean, why do GM cars still sell, even though they're just buckets of bolts that break down all the time? Apparently marketing is all that counts, because people aren't smart enough to put 2 and 2 together.

 

[Reverend]

Well, all I know is, if I ever do any video card reviews, I'll use something other than 3dMark to compare cards. It may be considered the industry standard but I'm sure there are other ways to get non-biased benchmark results.

Your statement implies that 3DMark03 is the only thing used in a video card review.

This is silly -- I don't think I have ever read a single video card review anywhere that only consists of using 3DMark03 (let's forget the print media) -- in most online reviews I have read, 3DMark03 is used in conjunction with other benchmarks.