Mr. Ojala

[YeuEmMaiMai]

first off to correct some errors in the math of the pdf

1. when figuring out a percentage of drop you have to do the following take the resulting differnce in the score and divide it by the original score so in this case

5806-4679=1127 now to find the percentage of drop you can do the following 1127/5806=19.41% so in actuality their score only dropped 19.41%

That is almost 6% less than what futuremark claimed the same can be applied to what ATI did

but the pdf does not give us the numbers so we cannot recalculate the score difference accurately for ATi.

How futuremark came up with their 24.09% was to take the "correct" score and divide it by the difference

4679/1127=24.09% However futuremark incorrectly stated that the scores dropped 24.09% when then should have stated that the cheats resulted in a 24.09% artificial increase of the score.


To: Futuremark

The release of Nvidia Detonator 44.03 drivers contained application specific cheats that gained them a 24% increase in performance for Nvidia FX cards on 3dmark03 build 320. This certainly raised some suspicion at ExtremeTech so they along with Beyond3d, began to investigate and reported their findings to you. You subsequently released a 7 page white paper in PDF format stating that Nvidia cheated. The PDF outlined exactly where and how they cheated, why this is bad, and the implications that may become of it. On June 2nd 2003, you issued a statement stating that nVidia did not cheat but instead has only done "application optimizations".

It was also announced that Nvidia and Futuremark have formed a new partnership.

Obviously )this is in the eyes of a consumer) cheating by altering the rendered output of your benchmark degrades your benchmarks validity. Cheating by ignoring commands, inseting clip planes, insertig sub-par shader code makes any compamy doing it look bad.

Although you may not have intended this to be the case, it appears that Nvidia has either:

1. Paid off Futuremark.

or

2. Threatened legal action as evidenced by this statement by Patric Olaja

" First I must admit that there is very little I can comment about the joint statement between Futuremark and Nvidia due to legal aspects. "

as posted on Tue Jun 03, 2003 9:10 am

http://www.beyond3d.com/forum/viewtopic.php?t=6230


We, the consumer, have just lost a lot of faith in Futuremark benchark as a valid benchmark. We the consumer demand benchmarks that accurately reflect the performance of hardware that claims to be DXn compliant.

Sincerely,

So what do you guys think?

 

[DethWraith]

Well, all I know is, if I ever do any video card reviews, I'll use something other than 3dMark to compare cards. It may be considered the industry standard but I'm sure there are other ways to get non-biased benchmark results.

Your statement implies that 3DMark03 is the only thing used in a video card review.

This is silly -- I don't think I have ever read a single video card review anywhere that only consists of using 3DMark03 (let's forget the print media) -- in most online reviews I have read, 3DMark03 is used in conjunction with other benchmarks.

That's not what I was implying at all. I was merely stating I would no longer use it as part of a review at all. In fact, I never really liked 3dMark all that much in the first place. I can't play it... I'd rather use games that people want to see benchmarked. However, it is supposed to give us a good idea of how future games would run on the hardware... that's where it has something over current games. With all the current problems going on, I could see a lot of graphics card companies taking advantage of this win for nVidia and using their own "optimizations" for 3dMark. This puts 3dMark in a very strange place... can it be trusted?

You do have to admit, however, that a lot of people base reviews off the 3dMark score. I never have, since I see benchmarking games as far more important, but if you ask some of the more uninformed people out there they would probably swear by 3dMark. That's the problem... we have to remember the average Joe doesn't know as much as some of us do and may look at reviews just for the 3dMark score. In past years many companies and review sites have stressed the importance of 3dMark, making it the industry standard benchmark program it is today. Why wouldn't people trust it, and only it?

 

[Reverend]

Then the question becomes :

How interested or concerned are you about technologies made available by the current latest video cards, which has proven to be not seen in any games you can either buy or download (for free or not) now? When the first TnL cards came out, did your purchase of such a card depended (even slightly) on its TnL feature, which you couldn't find in any existing game at the time? What about shader-featured (DX8) cards -- did your purchase of such a card depended on this shader feature, which again you'll never find in any existing game at the time? Or did you, when reading all those reviews, simply overlooked nor cared for 3DMarkXX, when you were looking for an upgrade?

Will you, or have you ever, considered a video card that you are looking to purchase based on how it will perform using the very features it advertises as "new" but which you know no game now uses? Can you honestly say you don't care about the new features of a new video card and how that video card performs when it has to operate using those new features as exemplified by 3DMarkXX? If you can tell me in my face that "No, I don't care about DX9 features of DX9 cards, I only care about how it performs using current available games", then, fine, I'll accept that... but I won't believe you.

But if you do care about the new features of a video card that you know will never be seen in any currently available game, what, pray tell, will you use as a measure of performance of such new features?

Everyone can say "I don't care about DX9 features of DX9 cards, I only care about how it performs using current available games, even if I know there are no DX9 games now" -- they're just lying because interest in new features, and the performance using such new features, is human nature.... you cannot resist it, you will definietly be interested.

 

[DethWraith]

Actually, when the first TnL cards came out, I bought a Voodoo 5, so that shows how much I cared about TnL over the visual quality of the Voodoo 5. However, we all know what the vast majority of people chose, and 3dfx stands as somewhat of a martyr of what good PR for one company can do to another. But yes, I was interested in new features, which both the original Geforce and the Voodoo 5 delivered, so you got me there.

My next card was a Radeon 7500, which I bought for the good price, I didn't really care about the 8500 at that time. I looked at various reviews for the performance in current games that I played, not at the 3DMark scores which reflected future hardware. At that time, I wanted something that performed good now, not something that would be good in the future. If I needed something in the future, I'd buy it when I needed it. I followed that purchase with an 8500 a few months later because a friend was getting rid of his and I got it for a good price. I'll admit, the first time I fired up Morrowind with that card, I stood there and looked at the water for a long time... shaders are cool. But, with the purchase of the 8500, I'll admit I started looking toward the future and 3DMark was a way to do this.

Anyway, when purchasing my 9500 I went for the price, and the fact that I could SoftMOD it to a 9700 , but, ya, this card is full of features which I like. Most are not being used since it is a DX9 card, so there's probably stuff this card can do that I don't even know what it is, unless I use 3DMark03 to see. To be honest, I was more than happy with my 8500... the only reason I jumped at the 9500 so quickly was because I heard they were going to modify the design so that the card could no longer be modded. Ya, I'm cheap.

As to what I'll use instead of 3DMark03... I couldn't tell you. To be honest, I didn't really think about it until you asked the question!

I'll admit, I'm interested in the features of new cards, to a point. I also look heavily at price/performance, which I think is the most important thing for people without a lot of money to think about when getting a new card. I also look at performance in current games more than performance in 3DMark, although this means I'd have to buy new cards when newer features are used in newer games. So, 3DMark is useful in showing what current graphics cards can do with future games... there's no doubt about that. I just find this whole nVidia situation confusing, since we're stuck with the question... does 3DMark really show us how current hardware will run future games? Sadly, only time will really tell.

 

[Reverend]

So, 3DMark is useful in showing what current graphics cards can do with future games... there's no doubt about that.

No, 3DMark is not useful in showing what current gfx cards can do with future games. 3DMark is useful in displaying, first and foremost, advanced features as made available by current cards (you gawk at what is being displayed in the various 3DMark tests first), and then secondly, the performance when video cards are performing these advanced features (you then look at the performance after gawking at the display). IOW, 3DMark is useful primarily in disseminating information to the public about what can be displayed when advanced features are used (surely you go "Gee, wonder what all these DX9 features looks like on screen?" before you go "Gee, wonder what the performance is?"). That is its major impact. 3DMark is a suite of technology demos utilizing new advanced features, with the added feature of being able to benchmark such technology demos.

I just find this whole nVidia situation confusing, since we're stuck with the question... does 3DMark really show us how current hardware will run future games? Sadly, only time will really tell.

Like I explained above, it is just an approximation. I am not comfortable with Futuremark's "The Gamers Benchmark" description/mantra for its 3DMarkXX software. As a beta member, I will try to convince FM that this should be replaced with "The Premier Advanced 3D Technology Benchmark" (or some such... perhaps even "The Hardware Geeks Benchmark" ) for the next 3DMark.

3DMarkXX will never be able to try to approximate correctly how future games will run or look like when using the same features as used in 3DMark's various tests. The history of the 3D industry is that no one game will feature all the new features as introduced by DX9 and DX9-compliant video cards. Games tend to feature a bit of this technology and a bit of another type of technology because the developers, with its creative team, have specific targets in mind, not only for artistic reasons but performance considerations. This extends beyond the PC and into the consoles -- even Splinter Cell on the XBOX never took advantage of every feature of the NV2A chip powering the XBOX.

The "Mother Nature" test of 3DMark03 is an extreme test (and the prettiest Game Test in 3DMark03) but should not be thought of as being representative of an actual game anytime soon. It involves, all at once, high dynamic color range (the sun), vs_2_0 sincos instruction for individual leaf and grass blade movements, multiple texture stages using ps_2_0 for the lake, cube maps, bump maps, light maps, color map, detail map... the list goes on. "Mother Nature" is an impressive display of a whole bunch of DX8 and DX9 features/technologies... and expectedly, performance will not be too impressive even with the latest/fastest video cards. The test shows you what happens (display, performance) when all these features/technologies are used. It does not show you that future games will be like that or even may be like that because Futuremark cannot "tell" game developers "Make your games look like this". That may sound silly but the point I'm trying to make is that Futuremark and game developers do not think alike nor have the same considerations, priorities and creativity.

So... what is 3DMarkXX good for? It is extremely good for showing you what the use of a multitude of advanced features will result in on your screen, first and foremost. The fact that we can benchmark a host of its tests is an added extra bonus and a relatively important one insofar, but not exceeding, as knowing what the performance would be like when such advanced features are used all at the same time but only matters in 3DMarkXX, and only 3DMarkXX, alone.

I support Futuremark and its 3DMarkXX suite of software because of these reasons. I don't particularly care for any public relations associated with it -- the "Gamers Benchmark" mantra is like a desciption of a game used by a game developer to advertise its game, like "The Future of First Person Action is here". I have relayed my disappointment about the recent FM/NV press release to certain Futuremark personnels. I can't tell you if they agree with me or not because that would not be right. I can tell you, however, that I tend to wear my heart on my sleeve and at times this tends to make things difficult for Dave and Beyond3D's industry partners. But I'd like to think this is good for the site.

Sorry if I'm rambling... I'm leaving for a weekend holiday and just felt like, well, rambling before I take leave

 

[DethWraith]

Good points. I'm not gonna debate anything you said, cuz there's nothing to debate. However, events of late have made 3DMark seem quite vulnerable to cheats, which has given it a bit of a bad rap. We'll just have to hope nothing like this happens again.

Btw, it's ok to ramble. I've been rambling too, and it's been fun. Nice rambling with you.

 

[CorwinB]

3dfx stands as somewhat of a martyr of what good PR for one company can do to another.

3dfx didn't disappear as a result of Nvidia's PR alone. Yes, Nvidia PR and their black ops guys (the "guys with webpages") played a big role there, but IMHO, 3dfx's business model, bad execution, and stoopid management also have their share of responsability.

"Never attribute to malice something that can be explained through stupidity."

 

[YeuEmMaiMai]

I think you guys should go read the front page of www.tech-report.com. Pat you might want to start looking for another job man as nVIdia got what they wanted...they have made your benchmark look like a easily manipulated peice of software..... whom ever analyzed the nvidia drivers and found those cheats missed one very important part... application name detection.....nVidia was screwing around with the ansio filtering and you guys did not catch it?

unfortunately for futuremark the appearent backing down from nVIdia and then this makes you guys 100% irrelivent as a benchmark tool

Futuremark should have stuck to it's guns but you folded instad and now there is a real possibility that your company will cease to exist.

I got one question that no one here has asked you

Since you have allowed nVidia to squeak by with those "application specific optiminzations" that reduce image quality how do you think this will affect the 9X standards since it is ok to optimize by itroducing techniques that lower IQ in favor of speed even when an application does not request the use of lower percision math?

 

[ET]

If they lose marketshare when they produce the best parts, what's going to happen now that they're back to being second best?

Well, AFAICR they gained market share, even though just a little, so the question is meaningless.

Besides, the low end is where the game is, and while ATI had a good high end and mid range part, NVIDIA still won the low end game with the 4MX.

 

[demalion]

Another example, based on "3dmurk"

...Continued from prior discussion

By having the label "Optimal", FM is representing IHV interests by allowing them to vary aniso implementations with more flexibility. I think that is fine as long as there exists some rules to limit that leeway in the interests of comparisons and to maintain user awareness of what is being done.

The anisotropic filtering option, the anisotropic filtering test, and rules for application detection would achieve that, but it depends on either the IHVs not implementing deceptive representation to hide issues. Also, it depends on providing clear information and/or tools for exposure of issues for both when they do that and when they do not. For example, it is important to have the aniso tool in 3dmark 03 to help expose the problems, or lack of problems, with things like angle variance in Radeon aniso, or issues with nVidia's "adaptive" aniso approaches, as long as misrepresentation isn't engineered by IHVs.

I think the combination of functionality exposure in the benchmark and information disclosure in press releases are part of this equation as well.

...

At current, I think the rule enforcement is lacking, and therefore the misrepresentation opportunity is made more prominent. I believe the information provided to users at current was sacrificed to trade recognition of IHV interests for behind-the-scenes decisions in favor of rule compliance from that IHV (to hopefully better serve users being informed in the future). The problem, however, is that the future isn't here yet, and until it is, the equation is "unbalanced", and users are done a disservice. You might have assurances that this is not the case, but users are not privy to the same degree of assurances.

From these and similar thoughts, as I've expressed before, I think functionality tools for exposure of issues can help address these problems, and also that new tools to analyze AA and aniso features and test for application detection should be part of the focus going forward. However, I also think there are some short term solutions in this regard at your disposal that should be considered for wider exposure to make the equation more "balanced" right now. I think increasing the minimum exposure of tools (I don't mean tools like publishing reports, just specifically tools focused on exposing information manipulations) for all users in some way is an alternative to "bald" statements, but you can't be lacking in both without doing users a disservice, and also that this consideration should be part of the long term evolution of benchmark tools.

Just some more input, if anyone is reading.