Another side of the story

[David G.]

I think 3DMark2003 is a real revolution from 2001 version as is synthetic from every point of view . Using game-like test engines in benchmarks makes that testing application be something hybrid .

Synthetic benchmarks , especially for graphics cards are very useful but only in FutureMark's last generation of software . Why ? Because it shows how much the additional HARDWARE optimization can technologies can boost the performance .

There you have it : you multiply the frequency of the memory with the bandwidth ... etc. and get the REAL bandwidth of the card . But hey , there comes the "greatest blablaest" video card manufacturer that says it's cards can show a bandwidth of 48 Gb/s or even 100 Gb/s because they've implemented "who knows what" technology that compresses the data and by multiplying with the compression factor we get that huge claimed bandwidth .

REAL synthetic benchmarking applications could help us see whose compression/optimization technology is better .

What I mean by "REAL" ? Well ... when 3DMark2001 was released I often wondered if that MAX Pain like engine isn't optimized for speed rather than quality . Every game maker want it's game to be available to a wider range of users with more or less powerful graphics solutions so they make some ways of optimizing/simplifying the rendered graphics so that the less powerful GPUs could get some speed .

Now that's software optimization , not hardware soooo ... how could we get an clear idea of ATi's HyperZ or nVIDIA's LDA if even a synthetic benchmark that is supposed to be able to do that is twisted around by software optimizations ?

When 03 was released I thought it was the perfect benchmark to test such technologies : you have 1Gb of textures to pass through your VPU and your card gets 1.2 Gb because it compresses some more data from the next step . Now that HARDWARE optimization .

There is another situation : from that 1Gb , your card renders only 400 Mb and so it's faster . That's software optimization .

We must not mix that .

REAL synthetic benchmarks are very important for graphic cards because , in most if not all the cases , the graphics cards don't reach even the theoretical peak performance .

Real applications represent a whole other chapter of testing graphic cards and we shouldn't say that synthetic benchmarking doesn't matter .


Now here's my pool . I have a question , why did FutureMark did all this sharade ? The way of putting this question is very important and I haven’t noticed any thread to put the problem in this way .

Cheater have always existed and there are more types of cheaters :

-ATi : trying the wares with the Quak thing which wasn't such a big deal but nVIDIA fans really inflated the whole situation ;

-SiS , Trident : they do this for a living , if not , their cards would be much too slow to make an impression on the customer ;

-nVIDIA : rising the cheating galore to unimaginable levels .


Why are nVIDIA's cheats so much more disturbing that the others ... ?

Well ... ATi's R200 was roughly twice as powerful as the R100 . SIS's Xabre was a whole revolution from the SIS 315 line .

nVIDIA's 5900 is ... well it only has some more bandwidth and less features .. clock speed etc.

But hey , the benchmarks show almost twice the performance is some situations .

That's why this whole thing is disturbing , because companies can get money from launching a "new" line of cards which's performance boost is based on cheating drivers .

That's dangerous .

So cheater were here for a long time but FutureMark only started to bite from nVIDIA's NV35 which ... is a lot better than NV30 from the architectural point of view and deserves it's money ( up to a point ) unlike the NV30 implementation .

There are two possibilities :

1) FutureMark decided to end , once and for all , cheating , and defend user's interests .

2) FutureMark decided that is they make enough scandal , nVIDIA will get back in the "Beta Program" officially or unofficially with a lot more contribution than before .

It is probably clear that if you design apps for the video card market and the biggest player in this market renegates you , you are put in a ... lets say "less credible position" .


Please give an objective answer and don't give in to subjectivity . Don't imagine that this thread insinuates FutureMark guys are the negative side of the story . Maybe they really are on to doing something better as they've started with the 03 generation .

I've written all of this because I'm horrofied by the cheating going of in he IT industry these days . After reading that ASUS sends much batter mainboards to reiewers than it sells in it's retail chanel and that MSI overclocks it's systems without letting us see it I think nVIDIA's "biggest lie " as JC called Geforce 4MX is getting small considering that it started a trend with ATi calling the RV250 9000 and the RV350 9600 .

Also , notice how ASUS calls it's line of NV3x cards "9800" in the 9700 Pro days and now "9900" just to make it sound bigger than ATi's 9800 Pro .

This is getting too dangerously close to efectively fooling the customer and stealing it's money .

And , also , please say if you agree with my wannabe definition of synthetic banchmarking in video card industry .

[Sharkfood]

A couple things-

There is no way to provide an "objective" answer, as you have requested, when the result is nothing more than speculation. Anytime you ask folks to speculate or form opinions, the end result will always be subjective.

Second, I'm a little confused towards the basis of your poll. Should the readers rewind time a few days and forgo what Futuremark has done in the recent future (i.e. reverse their "cheat" definition) and simply focus on why there was a debacle in the first place?

Well, unfortunately, it wasn't Futuremark that began the whole thing so therefore no speculation can truly be placed on their shoulders. Extremetech and some other websites are the ones that uncovered the issue and several other people performed research and work to try and determine/isolate the behavior. All that Futuremark did was release a patch... and a white-paper to document/define the outcry that was unveiled by the public.

If Futuremark had discovered, publicized and championed the situation, only then would I believe could you give credit (and as you have done, try to define motive) to Futuremark.

[martrox]

This is BS........ Plain & simple.....

[K.I.L.E.R]

Let me copy/paste something I told Reverend:


Assuming anything regardless of how obvious it is can be foolish in most situations especially if you don't know the full story.

Better ask before I drop a conclusion like many have done. It's already been answered on Mr Ojala's forum.


My point is that we may never know the full story. To assume anything or even provide feedback on the entire fiasco without knowing the full story is foolish.

People can comment on what may seem obvious but what may seem obvious might not be true at all, hence it is useless to truthfully comment and make judgements on these things.
So far I have read around 6 different theories to what's going on and only God, Futuremark and nVIDIA know what's going on.

We may never know what is really happening. We can only speculate and pray what we speculate is true.
Of course we must realise that the corporations will never give us the total truth and may bend the truth to suit their wants.

Of course it would sure as hell surprise me if those were bugs in NV's drivers. :)

[Humus]

There doesn't seem to be any poll option that applies, so I didn't vote.

[Doomtrooper]

I do think you will see this....

1) Nvidia will Rejoin the Beta Program

2) There will be about a month wait

3) There will be a patch from Futuremark (lets call it Patch 340) that will allow FX cards to use the DX9 _pp hint.


If I'm wrong I will eat my shoe. :lol:

I check the Beta program membership daily waiting for that Logo to appear :)

[CorwinB]

3) There will be a patch from Futuremark (lets call it Patch 340) that will allow FX cards to use the DX9 _pp hint.


New logo :
Nvidia : the way it's meant to be benchmarked.

[K.I.L.E.R]

If I'm wrong I will eat my shoe.


I will hold you to that. :lol:
You will have to provide pics too. :lol:

[Sharkfood]

I disagree Doomtrooper... but only on the part that stipulates NVidia will rejoin the beta program.

NVidia has already proven they don't need to join the beta program nor pay FM any fees at all to have their way with them.

[g__day]

Bad options - humus was right - poor choices to vote on.

I can't see FutureMark had any options if NVidia said welcome to war of the worlds. I think FM should side up closer to Dell to try and maintain their independence.

The moment they allow any videocard marker to fall beneath the DX9 spec or take shortcuts - we the consumers are being fooled.

If they can't honour us and our purchasing their software as in dependent software tools then their future looks compromised badly.

[David G.]

There doesn't seem to be any poll option that applies, so I didn't vote.

Ok ... so you think is not 1) and neither 2) .... give a little explenation .

You work for nVIDIA (?)... so you should have some insider info ...

[Doomtrooper]

I disagree Doomtrooper... but only on the part that stipulates NVidia will rejoin the beta program.

NVidia has already proven they don't need to join the beta program nor pay FM any fees at all to have their way with them.

True about the manipulation..we'll just wait and see I guess :D

[Doomtrooper]

There doesn't seem to be any poll option that applies, so I didn't vote.

Ok ... so you think is not 1) and neither 2) .... give a little explenation .

You work for nVIDIA (?)... so you should have some insider info ...

Humus doesn't work for anyone, he is a student but did spend some time working @ ATI, Toronto.

[WaltC]

As others have stated here your poll questions are the equivalent of asking a "loaded" question, so I also had to decline to answer as neither of the possibilities you list fits the evidence as I see it.

The problem for me with FM's current position is its logical inconsistency. They have now decided that "nVidia did not cheat" and yet they simultaneously state that the original audit report which clearly proves nVidia cheated stands as written. You can't say "nVidia didn't cheat" while also saying "nVidia cheated." Hopefully, the bright folks at FM will look a little deeper and understand that the intentional insertion of clip planes, buffer overrun artifacts and non-rendered frame segments do not and cannot ever equate to "application optimizations" as their audit .pdf points out so clearly. Further, there exist no 3D games in which nVidia employs these techniques as "application optimizations." Hence, nVidia's third claim (the first was "driver bugs," the second was "FM is picking on us intentionally") of "application optimizations" has no foundation in facts which nVidia has demonstrated.

Clearly, there is some division within FutureMark as to whether the benchmark should stand on its own as a defensible work of software with an intrinsic value, or whether it should merely become a marketing tool which exists solely to satisfy the marketing requirements of 3D hardware manufacturers. Until this schism within the company is healed it would appear as if the company's position relative to this matter is hopelessly muddled. It's a shame, too, because the presentation of the initial .pdf audit was so compelling, well-written, and clearly documented that to this day nVidia has been unable to coherently rebut it, and ATi felt compelled to respond to it.

[Humus]

There doesn't seem to be any poll option that applies, so I didn't vote.

Ok ... so you think is not 1) and neither 2) .... give a little explenation .

You work for nVIDIA (?)... so you should have some insider info ...

I think there was some ugly things going on behind our backs. There's no reason why FM should step back like that unless they have been threatened in some way. I don't think it's FM that's the bad guy here, I guess they just didn't have much of a choice.

[WaltC]

I think there was some ugly things going on behind our backs. There's no reason why FM should step back like that unless they have been threatened in some way. I don't think it's FM that's the bad guy here, I guess they just didn't have much of a choice.

Honestly, I think it might be more accurate to say that some individuals in FutureMark (not all of them) *came to believe* they had no choice. I think others within the company felt their initial audit report was soundly based and eminently defensible, and that the company's overarching goal should be to defend its software against unfounded public attack by companies outside of its software program.

In the nVidia-driven press releases I noticed that nVidia likes to call these events a "public feud" between the two companies. I never saw any evidence of that from FutureMark at all. If there was a "feud" it was entirely nVidia-fueled, IMO. The facts are that nVidia quit the program and went public with its angst back in December, and nVidia wilfully and with premeditation incorporated the driver cheats exposed by the FM audit report. If FM was ever "guilty" of anything it was self-defense, which you would expect out of any software company which produces software it deems worthwhile.

But apparently there are people within FM who *believed* they had no other choice and have unfortunately decided that the integrity of their company and its products is of lesser value than the propsect of what they fear *might happen* if nVidia remains displeased with them and their software. I think that is an extremely unfortunate turn for FM at a time when they were doing so well and had presented such a compelling defense of their software against the unfounded and unsubstantiated attacks by nVidia. What's the saying--"Snatching defeat from the jaws of victory"...? One can only hope that the individuals within the company who wrote the first audit report can convince the more timid within the company that they have a product which is worth defending against unfounded and unsubstantiated attacks regardless of where they come from. Appeasement is a tactic which is virtually guaranteed to fail.

[BRiT]

I didn't see this option in the poll:

C) Nvidia threatened to run them out of business via their army of laywers and near endless cash reserves.

[David G.]

There doesn't seem to be any poll option that applies, so I didn't vote.

Ok ... so you think is not 1) and neither 2) .... give a little explenation .

You work for nVIDIA (?)... so you should have some insider info ...

I think there was some ugly things going on behind our backs. There's no reason why FM should step back like that unless they have been threatened in some way. I don't think it's FM that's the bad guy here, I guess they just didn't have much of a choice.


True . That's exactly what I think ... and ... in a way .... what I hope as I really would like to belive that the FutueMark guys are the good guys ....

But , I'm not too much of a trusting person .

[David G.]

I didn't see this option in the poll:

C) Nvidia threatened to run them out of business via their army of laywers and near endless cash reserves.

Here you go ... since more wanted this , HUMUS realy convinced me there really can be such a situation .

[David G.]

What ?!

Does this mean that nVIDIA is now free to continue cheating ? !!! I mean , ok , so they made FutureMark back down but I never thought they'd be doing it again so soon ... (http://tech-report.com/etc/2003q2/3dmurk03/index.x?pg=1)

Edit : Remember what they said about ATi's Quak issue or what they are saying about Trident now ?

[Arun]

Justice is fair.
Injustice is even fairer...

But I believe it is important to realize this issue got out of hand. It was a big issue, agreed. We absolutely had to be vocal about it, agreed. But *that* much? No, I don't think so.
And why that? Simply because 99% of nVidia is perfectly innocent. From my understanding, this is an initiative by a small part of the driver team - not even most of it.


Uttar

[jb]

Justice is fair.
Injustice is even fairer...

But I believe it is important to realize this issue got out of hand. It was a big issue, agreed. We absolutely had to be vocal about it, agreed. But *that* much? No, I don't think so.
And why that? Simply because 99% of nVidia is perfectly innocent. From my understanding, this is an initiative by a small part of the driver team - not even most of it.
Uttar

WTF are you talking about? This shakes the very foundataion that all benchmark scores use. It complete remove trust from reviewers and their tools. How are they to know that new Demos in q3 will show different scores? How many more cheats...errr optimizations are there? All of this by a company that says they dont optimize for specific games not to mention benchmark. No Uttar, this is a very very big deal. NV has reamed us and hurt are trust in the pass. This time they killed all of what was left. It does not matter if its only 1 guy..as it only takes 1 guy to hurt a companies reputation...

[andypski]

And why that? Simply because 99% of nVidia is perfectly innocent. From my understanding, this is an initiative by a small part of the driver team - not even most of it.

Don't you think that modern source code control techniques make it quite difficult to sneak things into a driver codebase without the other driver engineers noticing?

[sonix666]

andypski, believe me, when developing on a project, people don't tend to put valueable time in reviewing everything that another developer does. So, no, I doubt that it would be hard to sneak such suspicious code in the driver base.

[sonix666]

WTF are you talking about? This shakes the very foundataion that all benchmark scores use. It complete remove trust from reviewers and their tools. How are they to know that new Demos in q3 will show different scores? How many more cheats...errr optimizations are there? All of this by a company that says they dont optimize for specific games not to mention benchmark. No Uttar, this is a very very big deal. NV has reamed us and hurt are trust in the pass. This time they killed all of what was left. It does not matter if its only 1 guy..as it only takes 1 guy to hurt a companies reputation...
Hold your horses. Right now, all the issues evolve around 3DMark2003 and noone has yet detected cheating anywhere else. People on the forums of www.aceshardware.com even tested if renaming UT2003.exe and D3DDrv.dll would influence the scores in Unreal Tournament 2003. And they didn't. So either, the optimizations are general, or for that specific Unreal engine, or they have very ingenius hacks.

[andypski]

andypski, believe me, when developing on a project, people don't tend to put valueable time in reviewing everything that another developer does. So, no, I doubt that it would be hard to sneak such suspicious code in the driver base.

That's funny - what sort of projects are you talking about? Because on the projects that I work on a very active interest is taken in what is being checked in, and a large amount of valuable time is taken in reviewing additions. It's all part of the quality assurance process.

[Dio]

More to the point, do you not think there is some regular performance testing of the drivers where sudden large increases would get noticed? I can see the email chain now...

"Hey, performance went up 20% today! Who did that?"
<stony silence>
"Well, who cares!"

This never happens.

[Clashman]

I didn't see this option in the poll:

C) Nvidia threatened to run them out of business via their army of laywers and near endless cash reserves.

Here you go ... since more wanted this , HUMUS realy convinced me there really can be such a situation .

And look at where the votes are already.

[LeStoffer]

And why that? Simply because 99% of nVidia is perfectly innocent. From my understanding, this is an initiative by a small part of the driver team - not even most of it.

I'm not entirely sure what your point is, but even if it was only a small part of the driver team that put this into effect, it is extremely naive to suggest or think that the management at nVidia - and they are the ones who represent the company - was clueless about this happening or afterwards didn't care about the damage it may have inflicted on its business. Sure most at nVidia is 'innocent' - in fact I bet that the majority opposed this taking place afterwards - but the idea that the executive board at nVidia is prepared to downplay this internally as just some jolly chaps having some jolly fun is absurd IMHO.

[sonix666]

For some parts of the code we do code review. But that normally takes only one other person. Not the whole team. ;)

[David G.]

Here , here , big quiestion: Why hasn't anybody comented the Tech-Report test officially ?

Did nVIDIA silenced FutureMark for good or we should expect one 3DMark03 patch for every nVIDIA driver release ?

[Tahir2]

Here , here , big quiestion: Why hasn't anybody comented the Tech-Report test officially ?

Did nVIDIA silenced FutureMark for good or we should expect one 3DMark03 patch for every nVIDIA driver release ?

Futuremark have been silenced.

[Sage]

I suspect that most engineers at nvidia give a damn about their creations and would never want to see them ridiculed so badly for cheating. But, they also have to take care of themselves and their families which means getting a paycheck, which means being the managements bitch sometimes. we all have done that at some time, I'm sure. The people to blame are the management and possibly a very few engineers, the rest are just following orders because they have to or are kept out of the loop on these things.

[Doomtrooper]

I do think you will see this....

1) Nvidia will Rejoin the Beta Program

2) There will be about a month wait

3) There will be a patch from Futuremark (lets call it Patch 340) that will allow FX cards to use the DX9 _pp hint.


If I'm wrong I will eat my shoe. :lol:

I check the Beta program membership daily waiting for that Logo to appear :)

It is starting:

1) http://www.futuremark.com/pressroom/pressreleases/?081203

:lol:

Waiting for the new patch now..

[WaltC]

...
NVidia has already proven they don't need to join the beta program nor pay FM any fees at all to have their way with them.

This has been an observation of mine as well. The only answer I can come up with is that nVidia legal has pointed out nVidia would be in a dicey lawsuit position should nVidia not be a paying member and the company go out of business at a time when nVidia is in open and public opposition to them and their software. In such a position I don't see how nVidia could avoid paying damages. Being a paying member would isolate them from those charges regardless of what has transpired over the last 9 months.

Come to think of it, is FM a private corporation? If so nVidia might be rejoining, but as a silent partner with a direct stake in the company. Just guessing...

[WaltC]

It is starting:

1) http://www.futuremark.com/pressroom/pressreleases/?081203

:lol:

Waiting for the new patch now..

Heh...;) At least the body of the press release says "re-joined"....I thought from some earlier statements I read that they might be publicizing it with no mention of the fact that this is not nVidia's first up as a "partner"....;)

[Dave H]

I do think you will see this....

1) Nvidia will Rejoin the Beta Program

2) There will be about a month wait

3) There will be a patch from Futuremark (lets call it Patch 340) that will allow FX cards to use the DX9 _pp hint.


If I'm wrong I will eat my shoe. :lol:

I check the Beta program membership daily waiting for that Logo to appear :)

It is starting:

1) http://www.futuremark.com/pressroom/pressreleases/?081203

:lol:

Waiting for the new patch now..

Of course, the _pp hint should be used in any PS 2.0 shader where the extra bits of precision are not necessary. (Which is probably most of those in 3dMark03.) They are part of the DX9 spec, and any neutral DX9 benchmark should use them where appropriate.

None of this is to say that FutureMark wasn't completely right to expose Nvidia for cheating in all sorts of ways on 3dMark03, or shouldn't be continuing to release patches to detect and disable any new cheats targeting patch 330 (which are almost certain to exist--in fact, known to exist in at least one case). In fact, the use of the _pp hint in 3dMark03 is neither here nor there, considering in the time frame of this whole controversy, non-quadro NV3x drivers treated all PS 2.0 code as if the _pp hint had been used.

Point is, if it hadn't been used, it probably should have, at least in many cases.

[jb]

Problem is how do they do this an remain fair? Remember when 3dmark2001SE came out? They added the PS1.4 test but did not let that add to the current scores? I fell the same thing should be done here.

[Doomtrooper]

Exactly I remember Worm and Me having some good disagreements on here about the lack of PS 1.4 included in scoring, and repeatedly stated the reason was that it would make their Database of scoring invalid...let us see if the same applies here. :wink:

[Nazgul]

Problem is how do they do this an remain fair? Remember when 3dmark2001SE came out? They added the PS1.4 test but did not let that add to the current scores? I fell the same thing should be done here.

Personally, I'd love to see them implement a 3-stage pixel shader test. All three runs would render the same scene, 1) with all shaders using full-precision, 2) with all shaders using partial-precision, 3) with a mixture of each. Making it a part of the final score would be tough, but adding it in like the PS1.4 test in 3DM2K1SE would be nice.

[Neeyik]

If a patch that allows you to force the _pp modifier comes about, it's not going to be as part of the default benchmark. Don't forget that 3DMark03 has already been patched that permits non-default benchmarking that "balances out" the testing between NVIDIA and ATI cards.

[Dave Baumann]

If a patch that allows you to force the _pp modifier comes about, it's not going to be as part of the default benchmark.

Although I believe the _PP modifier should be used in the next 3D benchmark, it should NOT be used as part of 3DMark03 now - that would alter the scores in the Orb and that was the reason cited for not updating 2001SE with PS1.4.

[OpenGL guy]

If a patch that allows you to force the _pp modifier comes about, it's not going to be as part of the default benchmark.

Although I believe the _PP modifier should be used in the next 3D benchmark, it should NOT be used as part of 3DMark03 now - that would alter the scores in the Orb and that was the reason cited for not updating 2001SE with PS1.4.
Are you sure it would modify the results? I mean, in theory, it could but I don't think it would, if you know what I mean ;)

[Neeyik]

If a patch that allows you to force the _pp modifier comes about, it's not going to be as part of the default benchmark.

Although I believe the _PP modifier should be used in the next 3D benchmark, it should NOT be used as part of 3DMark03 now - that would alter the scores in the Orb and that was the reason cited for not updating 2001SE with PS1.4.
Note that I said "default benchmark" - 3DMark03 already has the option to force PS1.1 in GT2/3, which levels the playing field between GF4 and 8500 cards. However, any such submitted score is automatically classed as "non-default" and therefore wouldn't be used in something like the Hall of Fame. A _pp option, if at all used, would almost certainly be done in a similar manner.

[Mariner]

Are you sure it would modify the results? I mean, in theory, it could but I don't think it would, if you know what I mean

Ha! Good one - very subtle. :lol: