Futuremark & NVIDIA Statement

Discussion in 'Beyond3D News' started by Dave Baumann, Jun 3, 2003.

  1. Dave Baumann

    Dave Baumann Gamerscore Wh...
    Moderator Legend

    Joined:
    Jan 29, 2002
    Messages:
    14,079
    Likes Received:
    648
    Location:
    O Canada!
    Not long after Dell's notification of support for Futuremarks 3DMark03 and for the first time since the release of 3DMark03 both Futuremark and NVIDIA have issued a joint statement in an attempt to settle some of the differences that have arisen during that period and address the highlighted issues in Futuremarks recent report. The main points of the statement are as follows:<blockquote>Futuremark Statement
    For the first time in 6 months, as a result of Futuremark's White Paper on May 23rd, 2003, Futuremark and NVIDIA have had detailed discussions regarding NVIDIA GPUs and Futuremark's 3DMark03 benchmark.

    Futuremark now has a deeper understanding of the situation and NVIDIA's optimization strategy. In the light of this, Futuremark now states that NVIDIA's driver design is an application specific optimization and not a cheat.

    The world of 3D Graphics has changed dramatically with the latest generation of highly programmable GPUs. Much like the world of CPUs, each GPU has a different architecture and a unique optimal code path. For example, Futuremark's PCMark2002 has different CPU test compilations for AMD's AthlonXP and Intel's Pentium4 CPUs.

    3DMark03 is designed as an un-optimized DirectX test and it provides performance comparisons accordingly. It does not contain manufacturer specific optimized code paths. Because all modifications that change the workload in 3DMark03 are forbidden, we were obliged to update the product to eliminate the effect of optimizations identified in different drivers so that 3DMark03 continued to produce comparable results.

    However, recent developments in the graphics industry and game development suggest that a different approach for game performance benchmarking might be needed, where manufacturer-specific code path optimization is directly in the code source. Futuremark will consider whether this approach is needed in its future benchmarks.

    NVIDIA Statement
    NVIDIA works closely with developers to optimize games for GeForceFX. These optimizations (including shader optimizations) are the result of the co-development process. This is the approach NVIDIA would have preferred also for 3DMark03.

    Joint NVIDIA-Futuremark Statement
    Both NVIDIA and Futuremark want to define clear rules with the industry about how benchmarks should be developed and how they should be used. We believe that common rules will prevent these types of unfortunate situations moving forward.
    </blockquote>While Futuremark climbs down from the word 'cheat' to describe the irregularities noted, and addressed in the recent 3.3.0 patch, NVIDIA have moved away from the stand point that these were either 'bugs' or that Futuremark had intentionally sought to harm NVIDIA's scores. Clearly this appears to be somewhat of a 'save face' statement from both parties, before it escalated beyond control, however the wording of the document sounds as though it leaves the door wide open for NVIDIA and Futuremark co-operating in the future, which raises the question as to whether NVIDIA will rejoin Futuremarks Beta program.

    From the wording of the statement, and from discussions with NVIDIA personnel, it has gradually become more apparant that it seems NVIDIA's initial dissatisfaction with 3DMark partly stems from the fact that it only uses full precision shader in some cases, rather than a mix of precisions that the GeForce FX architecture supports, and get most performance out of. We've posed a few questions to a number of developers on the issue of support for multiple precisions and whether its something they want to deal with or not - look out for the Q&amp;A session with a number of industry luminaries here, soon.

    As for 3DMark03, following this I posed the question as to what version of 3DMark03 should be used to Futuremarks VP of Benchmark Products, Aki Järvilehto and he was adamant that the 3.3.0 patch was still the correct version to use as benchmark specific optimisations are not allowed as far as Futuremark/3DMark is concerned. Futuremark will post an FAQ soon.

    Update: Reinforcing AJ's statement 3DMark Producer's, Patric Ojala, has written a small Q&amp;A in our forum in relation to the joint statement. Here's a snip:

    <blockquote>Q: Does this mean what you called originally as "cheats" actually were acceptable "optimizations", and that you made a wrong decicion in releasing Patch 330 and the Audit Report?
    A: By the definition of our benchmark and process, the optimizations are not acceptable. 3DMark scores are only comparable if drivers perform exactly the work 3DMark instructs them to do.

    As earlier stated, we recommend using the latest build 330 of 3DMark03, with the 44.03 (or 43.51 WHQL) Nvidia drivers, or the Catalyst 3.4 ATI drivers. This way obtained 3DMark03 results are genuinely comparable as far as we know.


    Q: Does this mean that in the future you will not make patches for 3DMark03 (or 3DMark2001) in order to reveal cheating?
    A: We might release further patches to 3DMark03, if a need for preventing driver optimizations appear in the future.</blockquote>Read the rest here.
     
  2. biff

    Newcomer

    Joined:
    May 15, 2003
    Messages:
    5
    Likes Received:
    0
    How can the clipping planes not be a cheat? And how can it be a benchmark with manufacturer specific optimizations? Would those optimizations work for every game? Methinks futuremark just shot themselves in the foot. :?
     
  3. FUDie

    Regular

    Joined:
    Sep 25, 2002
    Messages:
    581
    Likes Received:
    34
    Application specific optimizations don't lower image quality. Application specific optimizations don't break when the camera leaves a set track. The are cheats. End of discussion.

    Unless NVIDIA really feels that lowering image quality is a valid optimization. Makes you wonder why they bother making "high-end" cards at all. If image quality doesn't matter, NVIDIA can join the ranks of SIS and Trident and be in good company.

    -FUDie
     
  4. just me

    Newcomer

    Joined:
    Feb 15, 2003
    Messages:
    135
    Likes Received:
    0
    Idiots!

    Now I hope NV sues them for libelous statements that they themselves have just admitted were wrong.

    I was behind FM all the way > not now. They pee'd all over themselves & just destroyed any credibility I had for them or their products.

    I don't know who made this decision, but they obviously have no legal knowledge at all.

    Edit: For the 'why' of my opinion, look at my post on page 19 & the Court Case I posted.
     
  5. duncan36

    Newcomer

    Joined:
    Aug 14, 2002
    Messages:
    173
    Likes Received:
    0
    When a S&P500 company with a pack of lawyers threatens to crush your company if you dont retract a statement you retract it. Dont know if thats what happened here, but its been known to happen/ :wink:
     
  6. BRiT

    BRiT (╯°□°)╯
    Moderator Legend Alpha Subscriber

    Joined:
    Feb 7, 2002
    Messages:
    12,504
    Likes Received:
    8,707
    Location:
    Cleveland
    Oddly enough, I think [H] actually got this one right:
    Translation VIA HardOCP: FutureMark reneges on previous statements and confirms NVIDIA was not cheating on their benchmark and NVIDIA will not take a legal action against FutureMark that would bankrupt them. All about the $, but that is just our opinion.
     
  7. Anonymous

    Veteran

    Joined:
    May 12, 1978
    Messages:
    3,263
    Likes Received:
    0
    Does "S&P" stand for "Shady and Pathetic"?
     
  8. LeStoffer

    Veteran

    Joined:
    Feb 6, 2002
    Messages:
    1,253
    Likes Received:
    13
    Location:
    Land of the 25% VAT
    Nope. Futuremark simply couldn't survive being at war with nVidia. You have to look at it this way: both got something out if the statement, Futuremark stay healthy in business, while nVidia get an embarrasing episode out of the way.

    Pure business, pure business.
     
  9. Bolloxoid

    Newcomer

    Joined:
    May 15, 2003
    Messages:
    191
    Likes Received:
    0
    Ouch. I can understand that Futuremark does not want to quarrel forever with Nvidia and that they are tired of the publicity battle, but making a complete about-face like this does not do good for their credibility.

    Smells like "legal advice".
     
  10. mczak

    Veteran

    Joined:
    Oct 24, 2002
    Messages:
    3,015
    Likes Received:
    112
    I think that's a stupid move of futuremark - it hurts their credibility IMHO. And, I'm still sticking to my opinion that nvidia would have NEVER got to court, simply because they couldn't win and even if they would be able to drive futuremark out of business they'd still have lost (because the OEMs wouldn't have liked that for sure).
    But OTOH it's obvious futuremark can't "fight" nvidia forever - they are still one of only two major graphic chip manufacturers. If nvidia would continue to tell everybody how useless and biased towards ATI 3dmarks03 is at some point almost everybody would believe it (hell half of everybody already believes it).
     
  11. Anonymous

    Veteran

    Joined:
    May 12, 1978
    Messages:
    3,263
    Likes Received:
    0
    Also, FutureMark is saying that nVidia did not violate the spirit of the benchmark. If nVidia can lower image quality, then everyone else can too. What's the f'ing point of having high-end video cards if the vendors won't show you high-end graphics?

    This makes me sick.
     
  12. Anonymous

    Veteran

    Joined:
    May 12, 1978
    Messages:
    3,263
    Likes Received:
    0
    An absolute disgrace! I'm furious and I feel so sorry for poor ATI in all of this. First TechTV and now this. They just can't get a break!

    Clearly Futuremark had to make friends with Nvidia or watch their benchmark die under a barrage of Nvidiotic critique. Their benchmark would have no validity for comparison purposes if all Nvidia owners simply hate the test because it shows that their cards underperform. Futuremark needs Nvidia and therefore had no option.

    It is probably true that "apples to apples" doesn't make much sense anymore these days because practically all games do in fact optimize for Nvidia's special ways of doing things. They perform horribly otherwise.

    This just shows that Nvidia has pulled a "Microsoft" on the graphics card market. Sad...
     
  13. just me

    Newcomer

    Joined:
    Feb 15, 2003
    Messages:
    135
    Likes Received:
    0
    Edited a knee-jerk comment I need to 'put my eggs in a row' before I proceed with.
     
  14. Anonymous

    Veteran

    Joined:
    May 12, 1978
    Messages:
    3,263
    Likes Received:
    0
    Sickening... So now it's perfectly okay to do whatever the hell you want in 3dMark to achieve higher scores. Really great benchmark idea. With this I won't mind seeing 3DM03 go away. What use is the benchmark considering this development? What trust is left in FM's credibility (nV lost theirs a long time ago in my book)? We'll probably see scores breaking the 10K mark now due to cheats "optimizations". I was on FM's side before, but I guess independent, unbiased benchmarks running the exact same code is a fantasy. Maybe they thought they had to do this for fear of legal repercussions, but it still does not bode well for the benchmark.
     
  15. biff

    Newcomer

    Joined:
    May 15, 2003
    Messages:
    5
    Likes Received:
    0
    I don't know... it seems that futuremark is toast either way then. This really seems to undercut the credibility of their benchmark.

    Unless they introduce an option for "benchmark" vs "optimized" and allow NVIDIA to make hay with the "optimized" part provided the output was functionally equivalent. Lots of checking code to make sure NVIDIA wasn't still cheating. Trust them?
     
  16. Anonymous

    Veteran

    Joined:
    May 12, 1978
    Messages:
    3,263
    Likes Received:
    0
    1 It costs money to defend yourself against an organized legal assault, lots of money, and it really doesn't matter if you are guilty.

    2 If you recall, it was Futuremark who first told Nvidia to get lost when Nvidia complained about the way it was designed. The reply from Nvidia, in the form of that driver gave a heavy indication that if it chose to do so, Nvidia could invalidate Futuremarks primary source of business.

    3 However that creates a certain amount of heavy handedness, could cost both companies money best spent elsewhere (like having more marketing folks tell us how much we need both their products lol ) and would generally not move the industry forward.

    I think this is best described as an industry spat, where two parties both demonstrate what they can do if they ignore the other. Its over now in really the best way possible, both sides return to the corners and the parties work more for industry advancement than who's legal department is more heavily staffed.

    Move on, there's quite probably nothing more to see here.
     
  17. micron

    micron Diamond Viper 550
    Veteran

    Joined:
    Feb 23, 2003
    Messages:
    1,189
    Likes Received:
    12
    Location:
    U.S.
    ..............pointless.
     
  18. antlers

    Regular

    Joined:
    Aug 14, 2002
    Messages:
    457
    Likes Received:
    0
    If you really read the statement, Futuremark really did not back down that much. They just stopped calling it a "cheat". They still call it something that does not belong in NVidia's drivers that they have a responsibility to prevent working.
     
  19. muzz

    Newcomer

    Joined:
    Mar 6, 2003
    Messages:
    61
    Likes Received:
    0
    Location:
    USA/MA
    "Its over now in really the best way possible, both sides return to the corners and the parties work more for industry advancement"

    Thats some pretty PATHETIC advice....

    Move on?
    You move on.

    Best way possible?
    Gimme a break with that.
    This was bullying to the Max, 3DFX was closed down by them, FM would be too.
    Thats the way I see it, and if lowering precision under the DX9 SPEC ( ummm a dx9 benchmark is SUPPOSED to be to SPEC), or not rendering the full workload is ok with you, then I have some Enron stock I would like to sell you.

    BAH!!! this is BS. :evil:
     
  20. Anonymous

    Veteran

    Joined:
    May 12, 1978
    Messages:
    3,263
    Likes Received:
    0
    What's the point of buying a benchmark that special-cases for different
    hardware?

    It's like comparing apples to oranges surely?

    If FutureMark do go through with this then as far as I am
    concerned 3DMark03 will be nothing more than a demo;
    a serious benchmark? Like hell it is.

    What's is needed is for someone like SPEC to create
    a graphics benchmark that is more game friendly than
    ViewPerf. SPEC at least has proper by-laws defining
    how benchmarks are run, what quality must be achived,
    what are legitimate optimisations, and how results
    should be submitted. Although that said I think Beyond3d
    should investigate all the other key benchmarks too.

    This whole fiasco reflects badly on ATI, Futuremark and
    Nvidia.

    As a 3d programmer this makes things more difficult
    for me, how can I trust any benchmark? How can I
    fairly evaluate the performance of competing hardware?

    Even writing your own benchmark is becoming increasingly
    more difficult, what shader 'optimisations' are going on in
    the driver? Am I getting the image quality I expect? Or
    does the driver 'optimise' when it thinks it can get away
    with it?

    Shame on them all.
     
Loading...

Share This Page

  • About Us

    Beyond3D has been around for over a decade and prides itself on being the best place on the web for in-depth, technically-driven discussion and analysis of 3D graphics hardware. If you love pixels and transistors, you've come to the right place!

    Beyond3D is proudly published by GPU Tools Ltd.
Loading...