About NVIDIA, Futuremark and HardOCP

Scali

Regular
Here follows a mail I sent to Kyle and Steve, regarding the editorial, and later also the remark about Futuremark allegedly changing the standard, regarding the Microsoft Game Advisor news:

A site like Hardocp has a responsibility to their readers, it is time you people start taking it, and provide some proper journalism.

The editorial that you wrote seems quite subjective, and is not very good journalism in my eyes. Therefore I feel obliged to mail you some issues that I think are wrong with it, so that you perhaps can post an update later...

The biggest issue is the fact that you have only contacted NVIDIA about an explanation for the decrease in score after the new patch...
Why have you not contacted Futuremark aswell?
You now have a rather distorted view, since you only heard the NVIDIA-explanation, which is filled with marketing sauce, and you have no clear picture about the FM patch, so you simply make some assumptions, that suit NVIDIA's explanation (NVIDIA handily exploits your lack of knowledge on the subject, have to hand them that).

Your assumption is clearly expressed by this line:

"It is apparent that Futuremark is doing something to the NVIDIA driver so that it does things differently in this newer version of the benchmark."

This assumption is completely wrong, it is probably instilled by clever NVIDIA-talk such as:

"This latest patch from Futuremark is yet another revision of 3DMark03 specifically designed to defeat our Unified Compilier Technology, which evaluates shaders and in some cases substitutes hand tuned shaders, but increasingly simply applies the run-time compiler to generate optimal code."
...
"This is yet another example of how 3DMark03 doesn't behave like a game - as a game developer would never specifically try to make their application run poorly or disable optimizations that produce the correct image while delivering better performance."

You need to read between the lines here... They are implying that Futuremark actively sabotages their 'compiler'... But, if you read their definition of their 'compiler' closely, you see that they mention "substitutes hand tuned shaders" as a feature of their 'compiler'.
Using this definition of a 'compiler', then yes, the statements that NVIDIA makes are right... But this definition of a compiler is wrong ofcourse. It doesn't actually generate code, let alone optimize it. It simply replaces it with a shader that is prepared earlier.

With this in mind, let's look at what the FM patch really does (as the earlier patch did aswell)...
The patch replaces the shaders used in 3dmark03 with new versions, which are different in only one respect: the registers have been 'shuffled'.
This means that the shaders will perform the exact same operations, in the exact same order, but only using different registers. Since all registers are equal, and the code still uses them in the exact same fashion, it doesn't matter which one you use. The only difference is that there will be a few different bits in each shader, namely in the register operand fields.

This 'sabotages' NVIDIA's 'compiler', since its detection mechanism now fails to detect the shaders, and it cannot perform the replacement. It has to fall back to actually compiling the shader at runtime, as it should have been doing in the first place.
This is also the reason why other cards like the Radeon and the Volari are completely unaffected by the patch. They always compiled the shaders at runtime anyway, and therefore the results are still the same, and performance is unaffected.

One could argue that shaders are also replaced in games, and that this is the most realistic scenario, but clearly Futuremark wants to focus on testing the performance of the hardware itself, not on the amount of cleverness of the driver division to boost performance in selected games, with arbitrary standards for rendering quality and correctness.

Futuremark actually has a good point here, namely... It's easy to test with games that are optimized, just run the games, and check the performance.
But how can one predict performance of games that have not been optimized?
That's what a synthetic benchmark is for.
This is also the most interesting benchmark for me personally... It defines the base-level of performance of the hardware. You can assume that you get at least this amount of performance in any software.
This is much more telling than some figure that may be reached in software that has undergone the same level of extreme (and yes, I find a difference of 1000 points extreme) 'optimizations', but tells nothing about how badly the lesser or non-optimized software will run, while this is by far the largest group.

To react on some of your quotes:

"Futuremark seems to have a whim about something and all of sudden the scores change drastically without any true explanation."

As I have said before, the scores only change on the products of one specific vendor. I think that the vendor should be doing the explaining here, not FM.
But the explanation they gave you was a rather clever and manipulative one. And you fell for it.

"I am supposed to read a press release and go, "Oh, so now after three quarters, 3DMark2003 is finally doing what it is supposed to." No specifics mentioned, no reason given, just "we made it objective now...""

No you aren't, as I already said, you should have contacted FM aswell, and reflect their side of the story in the article aswell. I believe that FM was quite clear in the PDF documents they released regarding 3dmark03 and driver guidelines over the past months anyway.

"I would like nothing more to see a benchmark from Futuremark that was great and would unite the world of hardware geeks. Futuremark has tremendous reach, an incredible brand penetration, and the ability to give their products to the end users for free. That simply is an incredible thing. Too bad their current benchmark is junk."

I think the benchmark is doing a fine job (especially if you look at all the separate test-results that it generates, not just the overall score).
I also think that articles such as yours are in the way of the unity of hardware geeks.
I also think that NVIDIA has a larger reach and brand penetration than Futuremark, which makes their marketing quite dangerous to Futuremark. NVIDIA has been actively trying to destroy the credibility of Futuremark since the first ATi card beat NVIDIA's offerings in their benchmarks. And gullible (I hate to say it, but it's the truth) 'journalists' such as yourself have been willing pawns in their hands.
Finally, I think their benchmark is not junk at all... I am a Direct3D developer myself, and I generally find the same trends in performance of my products on various hardware as their benchmarks indicate (obviously because my products have not received the special NVIDIA 'compiler optimizations' yet, just like Half Life 2 and Tomb Raider: Angel of Darkness indicated aswell, at first).
If I were to design and build a Direct3D benchmark, I have no idea what I should be doing differently from Futuremark, actually. The things they do make a lot of sense to developers like me, generally.

Perhaps you should study the differences between NVIDIA's hardware and the other DX9 offerings more closely, and figure out why NVIDIA really NEEDS their 'optimizations' to keep up with the competition.
A good start is this article: http://www.3dcenter.org/artikel/cinefx/index_e.php
There are also many discussions on the excellent forum of Beyond3d.

Also, read your own thoughts in the article...
"I do not much believe anything that I am told by NVIDIA anymore. NVIDIA has lost their credibility this year and that is not something that is easy to regain. They need to stop “PRing†us to death"

This is exactly what I said aswell, why did I say it? Because you don't practice what you preach! Just look at the assumptions you've made!

"Aside from all the cheating and optimization, 3DMark2003 is simply too narrow of a look at gaming to accurately represent the big picture."

You'd be surprised how accurate 3DMark03 really is, in a variety of scenarios.
Performance all comes down to a few things: "How many texels can I fetch and how many instructions can I process in a given time?".
Since 3DMark03 has various shader and texture tests, and also has some complete scenes with the latest technologies applied (such as dynamic shadows, skinning, per-pixel lighting with normalmaps, Depth-of-Field, to name but a few..), it paints a very detailed picture of the performance of DX9-hardware. It also did this long before any DX9-games were available, and virtually all DX9-games that have been released since, have verified the large gap in performance between ATi and NVIDIA that 3DMark03 has shown since the beginning (before all the cheating started).

The bottom line is this: NVIDIA has taken a different approach to graphics than its competitors, and this places them at a disadvantage in DX9-software. Futuremark had to pay heavily for being the first to show this problem, basically it had to pay for the fact that it was such a good benchmark, that brought out the truth about performance.
NVIDIA basically sabotaged the product by introducing cheats to try and close the gap in 3DMark03 (and who knows how many other titles. Don't you find it remarkable that ATi always performs well in every new DX9-title from day 1, while NVIDIA has to wait for the next driver update to 'solve' the performance-problems).
Why do you still insist that this is all Futuremark's fault, and that their product is faulty?
I agree that 3DMark03 is only useful when there is no big cheating going on in the drivers, but you cannot blame Futuremark for the cheating, after all, they are not the ones doing it! In fact, they are working hard to try and eliminate cheating, and maintain the integrity of the benchmark. Their effort should be applauded, and NVIDIA's efforts should be banished.
Why are you doing the opposite? And worse: why don't you inform yourself properly before making these judgements in public?

In the meantime I found two sites that DID give a proper report, and had FM's explanation on the patches...
Take a look at this:
http://www.theinquirer.net/?article=12641
http://www.xbitlabs.com/news/video/display/20031112181114.html

I hope you will actually do something about this, rather than just thinking you're above people like me and sites like this, and think you don't have to change anything anyway (it happens).

Even NVIDIA agrees with me now:
http://www.xbitlabs.com/news/video/display/20031114041519.html
The actual compiler is not disabled, only the shader replacer.
And clearly the shader replacer should not even be active during 3DMark03, and the updated shaders clearly lead to a failing shader replacer, which is not a deliberate action of Futuremark to disable some part of the NVIDIA compiler specifically, but is just a consequence of a failing cheat-mechanism due to the patch, exactly as Futuremark intended.

So, it seems that you should be asking NVIDIA for a new explanation...

I am looking forward to your updated article.

To conclude, I absolutely think you are right that the 'gaming experience' is the most important... But keep in mind that this is a highly subjective and personal thing. I hope you will make it clear that your reviews describe YOUR gaming experience. You may favour speed over quality, but others might not, for example.
And also make it clear that 'gaming experience' has two components: gaming and experience. Meaning that the experience depends on the game in question, thanks to the 'optimizations' done by various manufacturers.
I also hope that you will still give objective information, like framerates in games, or all results in 3DMark03 (the overall score is really not that interesting, the separate shader, fillrate and other tests tell a much more complete story). Because some people (such as myself) are still highly interested in this kind of information, and build their own opinions of the reviewed hardware based on this. I hope you will try to write reviews where there is enough information for us to still make this possible.

Keep up the good work,

Scali

======

Kyle never gave a proper response at all (Only a "Thanks for your thoughts.", which may aswell have been an automated response). So I decided to send the mail to Steve aswell.
He replied, but he tried to defend it all as just opinions... Basically it came down to these two lines:

"it is simply our opinion, on our site."
...
"I fully support your right to an opinion, just as we have a right to disagree with the current trend in benchmarks, you have a right to defend it. We can respectfully agree to disagree."

This is fine, except... Opinions shouldn't be in news-items, that is not proper journalism. News should be provided as objectively as posssible. There is a responsibility to your readers.
Besides, not everything was just opinion. There were also skewed facts or even lies.
I tried to point this out in my next mail:

And if this is just your opinion, something must have gone wrong, the article doesn't make it clear at all that it is just your opinion, it presents many things as truths. This is not proper journalism as I said, journalism is about trying to reflect all sides of a story as objectively as possible.
And on top of that, some of the things written there are simply lies or assumptions, without even informing with Futuremark...

And now again I read that you slag off Futuremark in the message about Microsoft. There is something severely wrong here...
You should be reporting the news, not adding your own subjective remarks...
Certainly not in the way you did...
It only shows that you do not understand anything about the whole Futuremark issue.
What did Futuremark say? They only allow approved drivers now. So this little remark:

“your system specs meet the system requirements of this game. To ensure your game will continue to be compatible, please run this diagnostic again tomorrow…we might change system ratings by thenâ€.

is totally wrong and out-of-place.
The system ratings never changed. The exact same routines are benchmarked, and the exact same formula is used to calculate the final score (and with any card other than NVIDIA, the results are identical with either version, please think and draw some smart conclusions). Only the way NVIDIA handled the benchmark with their drivers changed. Futuremark tried to disable the cheats once, but NVIDIA just kept going... Now they are disabled for a second time, but here is the difference: this time you can only submit scores to the database if you have a driver that Futuremark approved to be handling the benchmark correctly, so no cheating.
NVIDIA can cheat all they want now, but Futuremark will simply not accept the results in their database now, so they will not influence the Game advisor.

When will you realize that the Futuremark standard never changed? You said it again. If you can't see what's really happening here, even after my explanations and links to other sources of useful info, then I can draw no other conclusion than that you are not capable of understanding modern hardware and software technology, and you are not fit for your job. Please respect the responsibility to your readers!

Scali

======

The reply I got was this:

I could've sworn this was our site.


Feel to use your own site http://www.scali.eu.org/Main.html to post your own opinions, I will provide a link so people can come to your site and tell you how to run it and call you names.

======

Apparently these people don't care what their readers think, and don't care whether what they say is right or not.
After the replies got increasingly cocky, I decided to just give up, and told them I would publish the emails, "hoping that it will save at least some of the less experienced from falling for your lies, because apparently you feel no need to change anything about the garbage that is on the site."

So here you are. I hope that people will go to more trustworthy sites when they want to read about 3d graphics software and hardware.

Scali
 
Well, that's the typical [H] to you... Your email was very good, professional and to the point, polite yet firm, and you got sarcasm and stupidity as a result. That only further confirms that [K] and most of his crew are just acting like spoiled brats throwing tantrums when things don't go their way... As you noticed, even though [K] know Nvidia tricked him, he's still being used because he seems to lack the basic understanding of what's going on.

Feel to use your own site http://www.scali.eu.org/Main.html to post your own opinions, I will provide a link so people can come to your site and tell you how to run it and call you names.

And he very well may... Just like when he made public the emails of some people who had donated very low amounts of money to [H] so they would get mailbombed by the "Community" (using the term loosely here) at [H]...
 
CorwinB said:
And he very well may... Just like when he made public the emails of some people who had donated very low amounts of money to [H] so they would get mailbombed by the "Community" (using the term loosely here) at [H]...
Oh yeah, I remember this!
Funny thing, he violated his very own holy forum rules by that.
Talk about ntegrity.. :rolleyes:
 
If I ever respond to constructive criticism or valid, polite comments like that, can somebody please shoot me? Thanks. :p
 
http://www.hardforum.com/showthread.php?s=&threadid=691332To clarify, the journalism-part was mostly aimed at the following text, found in Hardnews, 4th edition, of Sunday November 16th:


Microsoft & Futuremark:
Microsoft Windows XP Game Advisor is a tool that is supposed to tell you whether or not your PC can run today’s latest games. Well it is a good thing that it is made by FutureMark…wait a minute, aren’t they the same people in our editorial? Interesting. Thanks to Matt E. for the linkage.


Once you've selected a game, make sure you're ready to play by clicking the "Will it run on my PC?" link. You'll find out whether your computer matches up with the game, technically speaking. A small applet from FutureMark Corporation will look over your computer and make sure the game you've selected will run smoothly on the computer you're using to browse the Web.

Heh…they should have a disclaimer “your system specs meet the system requirements of this game. To ensure your game will continue to be compatible, please run this diagnostic again tomorrow…we might change system ratings by thenâ€. That’d be pretty funny.

======

It is obviously called 'news', but it is obviously very subjective, and it refers directly to the editorial... As if it were news aswell? Parts of the editorial in fact ARE news, I'd say... The first part, containing the statements from Futuremark, and the reaction from NVIDIA, have not been published elsewhere on the site, and are presented here for the first time, and they are ofcourse not Kyle's opinions, but factual statements from these companies. Only the second part is actually an editorial, albeit a misinformed one, since the first part (the part where information is gathered) is not execucted correctly. Whether that is journalism or not, is beside the point. The point I was trying to make, is that if you have a popular website, I believe you have the responsibility to inform yourself properly before forming an opinion that you expose to so many people.
 
He seems to think that it's illegal for a piece of software to affect the driver. But he seems to think the opposite is just fine and dandy.
 
The Baron said:
He seems to think that it's illegal for a piece of software to affect the driver. But he seems to think the opposite is just fine and dandy.
Replace "think" with "have been informed" a couple of times and I can agree with you a lot easier. ;)
 
See, I don't even think there's some grand NVIDIA-[H] conspiracy. I think Kyle's simply an idiot.
 
Reading through that thread is depressing. It's like beating your head against a wall, but the depressing part is that the wall appears to be more intelligent as a whole than, say, Anandtech's forums. Some [H] regulars just deliberately (and shockingly short-sightedly) ignore the ramifications of nV's cheating, for reasons I can't understand. ambiguicon [sic] typifies the absurdly frustrating responses I see so often but am not yet inured to. He understands that nV is cheating, but instead of trying to fix the problem, he throws away the test and moves onto another. Why doesn't he realize that nV will do the same (move from cheating on test A to cheating on test B)? Didn't he see that nV's AM3 scores jumped 100% with two driver releases? Doesn't he know that nV's and ATi's miraculous speedups in 3DM01SE's Nature test were also caused by hand-coded shader replacements, something that neither IHV can afford to do for every game out there?

It's very frustrating. I wonder if this whole public debacle will repeat itself with 3DM04, or if FM will take the sadly realistic road and include custom codepaths (thus defeating the purpose of a high-level API).
 
Pete said:
It's very frustrating. I wonder if this whole public debacle will repeat itself with 3DM04, or if FM will take the sadly realistic road and include custom codepaths (thus defeating the purpose of a high-level API).

...and destroys the whole meaning/idea of this 3DM-series as well.
 
I don't agree with people who espouse the viewpoint "3dmark is a useless benchmark, so it doesn't matter if Nvidia cheats, only games matter".

Simply put, if I were to believe 3dmark was as absolutely useless as they say, then I would be even more opposed to Nvidia's optimizations. For every in game corrupt texture, unnecessary stutter, and compromise in stability, there is effort being spent by Nvidia on something entirely meaningless that might even contribute to driver failure.

It's not like Nvidia is only cheating on 3dmark, either, and has before compromised the image quality of popular benchmarkable games, or stealthily undermined the application behavior that the developers had intended and assumed to be stable during their support efforts.

3dmark03 itself has its limitations, as there may be a far larger number of systems that will run next generation games that will not be bottlenecked primarily by the graphics card, a consideration which does not seem to be considered as significant in the latest benchmarks. Despite this, to throw away a useful theoretical and experimental tool for probing the performance of a 3d accellerator is rather rash.

I do, however, see where some of them are coming from in their suspiciouns of futuremark's taking money from IHVs in its beta program. That all major companies are members at the moment partially negates that, though only insofar as that I trust there is a certain amount of distrust sufficient to hinder collusion amongst those involved.
 
Pete said:
Didn't he see that nV's AM3 scores jumped 100% with two driver releases?

Heh. I don't think I've ever seen as much circular logic or absurd justifications to emotional arguments as around this topic. While I tend to smile and shurg it off, it still is very sad.

One thing that I can't help but wonder is the fact that most people in the industry take it for granted that drivers can improve performance. So in general we accept that performance does change over time. Hardware _and_ software both influence this.

Now I'm reading arguments saying "how can we trust a benchmark if the scores change"? These arguments also at times come from people who have welcomed the "driver boosts" with new releases. I'm afraid a more valid question that we should have been asking all along would have been "how can we trust the drivers if the scores change".

I'm happy that Futuremark Crew is doing things about it. It's also nice to see that even in the [H] forums there are sensible people who care about these things. My hat's off to Scali who wrote an excellent piece as the discussion opener.

Cheers,

AJ
 
AJ, are you quoting me as an example of the wrong kind of logic/justification/argument? Seriously, I'm not sure. :) I think I have a good grasp on the implications (and ethics, tangential as they are in business) of nV's "work-arounds," but I've had my eyes opened more than once this year regarding optimal software/hardware. (I fairly firmly believe that Kyle has approached the DX9 issue wrongly, though. Even if his heart or mind is in the right place, his editorials don't come across as fully informed or unbiased.)

I was one of the many who were impressed with nV's inevitable and substantial driver speed-ups (the GF3's in particular). I have since become one of the (growing) few who have, since 3DM03's release, become suspicious of large speed boosts. FM put forward the new-to-me argument that initial drivers aren't going to be much improved upon with legitimate, universal optimisations. In seeming contradiction to that, MS then released the PS2a HLSL compiler/"path" (and the Det52 seemed to improve performance without sacrificing IQ). I can only conclude that driver boosts may not only be legitimate, but necessary, particularly with hardware as divergent as the current pair currently lumped together as "DX9." It's obvious that IHV's, like software devs, are releasing unfinished product ahead of the QA curve to begin to recover their costs early, with subsequent patches and even respins a fact of life. MS, OTOH, in an effort to make PC gaming as painless (up front, at least) as possible, has taken to naming both the FX and R300 architecture as DX9, though clearly the two have shown hugely divergent performance.

The question is, how do we incorporate these facts into benchmark testing? As someone pointed out to me, nV may well improve performance in Max Payne 2 with later drivers, but how does that help the guy who bought an FX card and MP2 at release? Can we honestly label a card as the best "DX9" card out there, knowing that, in the case of DX9.0, the cards seem to be stacked in favor of ATi's DX9 design?
 
[H] used to be a good link site. Now Driverheaven is better for that.

And the people at Driverheaven are not complete asshats.
 
Pete said:
AJ, are you quoting me as an example of the wrong kind of logic/justification/argument? Seriously, I'm not sure. :)

Sorry if I was confusing. Your logic is cool. :) I quoted you since you make a good point about Aquamark increases and I wanted to point out that for some strange reason people seem to accept even insanely high jumps upward in performance, while making a fuzz about decreses.

A 100% increase is great, but 15% decrease is 'unacceptable'.

The people can't be that clueless so one can only assume that they are separate agendas in play.

Cheers,

AJ
 
Back
Top