Here follows a mail I sent to Kyle and Steve, regarding the editorial, and later also the remark about Futuremark allegedly changing the standard, regarding the Microsoft Game Advisor news:
A site like Hardocp has a responsibility to their readers, it is time you people start taking it, and provide some proper journalism.
The editorial that you wrote seems quite subjective, and is not very good journalism in my eyes. Therefore I feel obliged to mail you some issues that I think are wrong with it, so that you perhaps can post an update later...
The biggest issue is the fact that you have only contacted NVIDIA about an explanation for the decrease in score after the new patch...
Why have you not contacted Futuremark aswell?
You now have a rather distorted view, since you only heard the NVIDIA-explanation, which is filled with marketing sauce, and you have no clear picture about the FM patch, so you simply make some assumptions, that suit NVIDIA's explanation (NVIDIA handily exploits your lack of knowledge on the subject, have to hand them that).
Your assumption is clearly expressed by this line:
"It is apparent that Futuremark is doing something to the NVIDIA driver so that it does things differently in this newer version of the benchmark."
This assumption is completely wrong, it is probably instilled by clever NVIDIA-talk such as:
"This latest patch from Futuremark is yet another revision of 3DMark03 specifically designed to defeat our Unified Compilier Technology, which evaluates shaders and in some cases substitutes hand tuned shaders, but increasingly simply applies the run-time compiler to generate optimal code."
...
"This is yet another example of how 3DMark03 doesn't behave like a game - as a game developer would never specifically try to make their application run poorly or disable optimizations that produce the correct image while delivering better performance."
You need to read between the lines here... They are implying that Futuremark actively sabotages their 'compiler'... But, if you read their definition of their 'compiler' closely, you see that they mention "substitutes hand tuned shaders" as a feature of their 'compiler'.
Using this definition of a 'compiler', then yes, the statements that NVIDIA makes are right... But this definition of a compiler is wrong ofcourse. It doesn't actually generate code, let alone optimize it. It simply replaces it with a shader that is prepared earlier.
With this in mind, let's look at what the FM patch really does (as the earlier patch did aswell)...
The patch replaces the shaders used in 3dmark03 with new versions, which are different in only one respect: the registers have been 'shuffled'.
This means that the shaders will perform the exact same operations, in the exact same order, but only using different registers. Since all registers are equal, and the code still uses them in the exact same fashion, it doesn't matter which one you use. The only difference is that there will be a few different bits in each shader, namely in the register operand fields.
This 'sabotages' NVIDIA's 'compiler', since its detection mechanism now fails to detect the shaders, and it cannot perform the replacement. It has to fall back to actually compiling the shader at runtime, as it should have been doing in the first place.
This is also the reason why other cards like the Radeon and the Volari are completely unaffected by the patch. They always compiled the shaders at runtime anyway, and therefore the results are still the same, and performance is unaffected.
One could argue that shaders are also replaced in games, and that this is the most realistic scenario, but clearly Futuremark wants to focus on testing the performance of the hardware itself, not on the amount of cleverness of the driver division to boost performance in selected games, with arbitrary standards for rendering quality and correctness.
Futuremark actually has a good point here, namely... It's easy to test with games that are optimized, just run the games, and check the performance.
But how can one predict performance of games that have not been optimized?
That's what a synthetic benchmark is for.
This is also the most interesting benchmark for me personally... It defines the base-level of performance of the hardware. You can assume that you get at least this amount of performance in any software.
This is much more telling than some figure that may be reached in software that has undergone the same level of extreme (and yes, I find a difference of 1000 points extreme) 'optimizations', but tells nothing about how badly the lesser or non-optimized software will run, while this is by far the largest group.
To react on some of your quotes:
"Futuremark seems to have a whim about something and all of sudden the scores change drastically without any true explanation."
As I have said before, the scores only change on the products of one specific vendor. I think that the vendor should be doing the explaining here, not FM.
But the explanation they gave you was a rather clever and manipulative one. And you fell for it.
"I am supposed to read a press release and go, "Oh, so now after three quarters, 3DMark2003 is finally doing what it is supposed to." No specifics mentioned, no reason given, just "we made it objective now...""
No you aren't, as I already said, you should have contacted FM aswell, and reflect their side of the story in the article aswell. I believe that FM was quite clear in the PDF documents they released regarding 3dmark03 and driver guidelines over the past months anyway.
"I would like nothing more to see a benchmark from Futuremark that was great and would unite the world of hardware geeks. Futuremark has tremendous reach, an incredible brand penetration, and the ability to give their products to the end users for free. That simply is an incredible thing. Too bad their current benchmark is junk."
I think the benchmark is doing a fine job (especially if you look at all the separate test-results that it generates, not just the overall score).
I also think that articles such as yours are in the way of the unity of hardware geeks.
I also think that NVIDIA has a larger reach and brand penetration than Futuremark, which makes their marketing quite dangerous to Futuremark. NVIDIA has been actively trying to destroy the credibility of Futuremark since the first ATi card beat NVIDIA's offerings in their benchmarks. And gullible (I hate to say it, but it's the truth) 'journalists' such as yourself have been willing pawns in their hands.
Finally, I think their benchmark is not junk at all... I am a Direct3D developer myself, and I generally find the same trends in performance of my products on various hardware as their benchmarks indicate (obviously because my products have not received the special NVIDIA 'compiler optimizations' yet, just like Half Life 2 and Tomb Raider: Angel of Darkness indicated aswell, at first).
If I were to design and build a Direct3D benchmark, I have no idea what I should be doing differently from Futuremark, actually. The things they do make a lot of sense to developers like me, generally.
Perhaps you should study the differences between NVIDIA's hardware and the other DX9 offerings more closely, and figure out why NVIDIA really NEEDS their 'optimizations' to keep up with the competition.
A good start is this article: http://www.3dcenter.org/artikel/cinefx/index_e.php
There are also many discussions on the excellent forum of Beyond3d.
Also, read your own thoughts in the article...
"I do not much believe anything that I am told by NVIDIA anymore. NVIDIA has lost their credibility this year and that is not something that is easy to regain. They need to stop “PRing†us to death"
This is exactly what I said aswell, why did I say it? Because you don't practice what you preach! Just look at the assumptions you've made!
"Aside from all the cheating and optimization, 3DMark2003 is simply too narrow of a look at gaming to accurately represent the big picture."
You'd be surprised how accurate 3DMark03 really is, in a variety of scenarios.
Performance all comes down to a few things: "How many texels can I fetch and how many instructions can I process in a given time?".
Since 3DMark03 has various shader and texture tests, and also has some complete scenes with the latest technologies applied (such as dynamic shadows, skinning, per-pixel lighting with normalmaps, Depth-of-Field, to name but a few..), it paints a very detailed picture of the performance of DX9-hardware. It also did this long before any DX9-games were available, and virtually all DX9-games that have been released since, have verified the large gap in performance between ATi and NVIDIA that 3DMark03 has shown since the beginning (before all the cheating started).
The bottom line is this: NVIDIA has taken a different approach to graphics than its competitors, and this places them at a disadvantage in DX9-software. Futuremark had to pay heavily for being the first to show this problem, basically it had to pay for the fact that it was such a good benchmark, that brought out the truth about performance.
NVIDIA basically sabotaged the product by introducing cheats to try and close the gap in 3DMark03 (and who knows how many other titles. Don't you find it remarkable that ATi always performs well in every new DX9-title from day 1, while NVIDIA has to wait for the next driver update to 'solve' the performance-problems).
Why do you still insist that this is all Futuremark's fault, and that their product is faulty?
I agree that 3DMark03 is only useful when there is no big cheating going on in the drivers, but you cannot blame Futuremark for the cheating, after all, they are not the ones doing it! In fact, they are working hard to try and eliminate cheating, and maintain the integrity of the benchmark. Their effort should be applauded, and NVIDIA's efforts should be banished.
Why are you doing the opposite? And worse: why don't you inform yourself properly before making these judgements in public?
In the meantime I found two sites that DID give a proper report, and had FM's explanation on the patches...
Take a look at this:
http://www.theinquirer.net/?article=12641
http://www.xbitlabs.com/news/video/display/20031112181114.html
I hope you will actually do something about this, rather than just thinking you're above people like me and sites like this, and think you don't have to change anything anyway (it happens).
Even NVIDIA agrees with me now:
http://www.xbitlabs.com/news/video/display/20031114041519.html
The actual compiler is not disabled, only the shader replacer.
And clearly the shader replacer should not even be active during 3DMark03, and the updated shaders clearly lead to a failing shader replacer, which is not a deliberate action of Futuremark to disable some part of the NVIDIA compiler specifically, but is just a consequence of a failing cheat-mechanism due to the patch, exactly as Futuremark intended.
So, it seems that you should be asking NVIDIA for a new explanation...
I am looking forward to your updated article.
To conclude, I absolutely think you are right that the 'gaming experience' is the most important... But keep in mind that this is a highly subjective and personal thing. I hope you will make it clear that your reviews describe YOUR gaming experience. You may favour speed over quality, but others might not, for example.
And also make it clear that 'gaming experience' has two components: gaming and experience. Meaning that the experience depends on the game in question, thanks to the 'optimizations' done by various manufacturers.
I also hope that you will still give objective information, like framerates in games, or all results in 3DMark03 (the overall score is really not that interesting, the separate shader, fillrate and other tests tell a much more complete story). Because some people (such as myself) are still highly interested in this kind of information, and build their own opinions of the reviewed hardware based on this. I hope you will try to write reviews where there is enough information for us to still make this possible.
Keep up the good work,
Scali
======
Kyle never gave a proper response at all (Only a "Thanks for your thoughts.", which may aswell have been an automated response). So I decided to send the mail to Steve aswell.
He replied, but he tried to defend it all as just opinions... Basically it came down to these two lines:
"it is simply our opinion, on our site."
...
"I fully support your right to an opinion, just as we have a right to disagree with the current trend in benchmarks, you have a right to defend it. We can respectfully agree to disagree."
This is fine, except... Opinions shouldn't be in news-items, that is not proper journalism. News should be provided as objectively as posssible. There is a responsibility to your readers.
Besides, not everything was just opinion. There were also skewed facts or even lies.
I tried to point this out in my next mail:
And if this is just your opinion, something must have gone wrong, the article doesn't make it clear at all that it is just your opinion, it presents many things as truths. This is not proper journalism as I said, journalism is about trying to reflect all sides of a story as objectively as possible.
And on top of that, some of the things written there are simply lies or assumptions, without even informing with Futuremark...
And now again I read that you slag off Futuremark in the message about Microsoft. There is something severely wrong here...
You should be reporting the news, not adding your own subjective remarks...
Certainly not in the way you did...
It only shows that you do not understand anything about the whole Futuremark issue.
What did Futuremark say? They only allow approved drivers now. So this little remark:
“your system specs meet the system requirements of this game. To ensure your game will continue to be compatible, please run this diagnostic again tomorrow…we might change system ratings by thenâ€.
is totally wrong and out-of-place.
The system ratings never changed. The exact same routines are benchmarked, and the exact same formula is used to calculate the final score (and with any card other than NVIDIA, the results are identical with either version, please think and draw some smart conclusions). Only the way NVIDIA handled the benchmark with their drivers changed. Futuremark tried to disable the cheats once, but NVIDIA just kept going... Now they are disabled for a second time, but here is the difference: this time you can only submit scores to the database if you have a driver that Futuremark approved to be handling the benchmark correctly, so no cheating.
NVIDIA can cheat all they want now, but Futuremark will simply not accept the results in their database now, so they will not influence the Game advisor.
When will you realize that the Futuremark standard never changed? You said it again. If you can't see what's really happening here, even after my explanations and links to other sources of useful info, then I can draw no other conclusion than that you are not capable of understanding modern hardware and software technology, and you are not fit for your job. Please respect the responsibility to your readers!
Scali
======
The reply I got was this:
I could've sworn this was our site.
Feel to use your own site http://www.scali.eu.org/Main.html to post your own opinions, I will provide a link so people can come to your site and tell you how to run it and call you names.
======
Apparently these people don't care what their readers think, and don't care whether what they say is right or not.
After the replies got increasingly cocky, I decided to just give up, and told them I would publish the emails, "hoping that it will save at least some of the less experienced from falling for your lies, because apparently you feel no need to change anything about the garbage that is on the site."
So here you are. I hope that people will go to more trustworthy sites when they want to read about 3d graphics software and hardware.
Scali
A site like Hardocp has a responsibility to their readers, it is time you people start taking it, and provide some proper journalism.
The editorial that you wrote seems quite subjective, and is not very good journalism in my eyes. Therefore I feel obliged to mail you some issues that I think are wrong with it, so that you perhaps can post an update later...
The biggest issue is the fact that you have only contacted NVIDIA about an explanation for the decrease in score after the new patch...
Why have you not contacted Futuremark aswell?
You now have a rather distorted view, since you only heard the NVIDIA-explanation, which is filled with marketing sauce, and you have no clear picture about the FM patch, so you simply make some assumptions, that suit NVIDIA's explanation (NVIDIA handily exploits your lack of knowledge on the subject, have to hand them that).
Your assumption is clearly expressed by this line:
"It is apparent that Futuremark is doing something to the NVIDIA driver so that it does things differently in this newer version of the benchmark."
This assumption is completely wrong, it is probably instilled by clever NVIDIA-talk such as:
"This latest patch from Futuremark is yet another revision of 3DMark03 specifically designed to defeat our Unified Compilier Technology, which evaluates shaders and in some cases substitutes hand tuned shaders, but increasingly simply applies the run-time compiler to generate optimal code."
...
"This is yet another example of how 3DMark03 doesn't behave like a game - as a game developer would never specifically try to make their application run poorly or disable optimizations that produce the correct image while delivering better performance."
You need to read between the lines here... They are implying that Futuremark actively sabotages their 'compiler'... But, if you read their definition of their 'compiler' closely, you see that they mention "substitutes hand tuned shaders" as a feature of their 'compiler'.
Using this definition of a 'compiler', then yes, the statements that NVIDIA makes are right... But this definition of a compiler is wrong ofcourse. It doesn't actually generate code, let alone optimize it. It simply replaces it with a shader that is prepared earlier.
With this in mind, let's look at what the FM patch really does (as the earlier patch did aswell)...
The patch replaces the shaders used in 3dmark03 with new versions, which are different in only one respect: the registers have been 'shuffled'.
This means that the shaders will perform the exact same operations, in the exact same order, but only using different registers. Since all registers are equal, and the code still uses them in the exact same fashion, it doesn't matter which one you use. The only difference is that there will be a few different bits in each shader, namely in the register operand fields.
This 'sabotages' NVIDIA's 'compiler', since its detection mechanism now fails to detect the shaders, and it cannot perform the replacement. It has to fall back to actually compiling the shader at runtime, as it should have been doing in the first place.
This is also the reason why other cards like the Radeon and the Volari are completely unaffected by the patch. They always compiled the shaders at runtime anyway, and therefore the results are still the same, and performance is unaffected.
One could argue that shaders are also replaced in games, and that this is the most realistic scenario, but clearly Futuremark wants to focus on testing the performance of the hardware itself, not on the amount of cleverness of the driver division to boost performance in selected games, with arbitrary standards for rendering quality and correctness.
Futuremark actually has a good point here, namely... It's easy to test with games that are optimized, just run the games, and check the performance.
But how can one predict performance of games that have not been optimized?
That's what a synthetic benchmark is for.
This is also the most interesting benchmark for me personally... It defines the base-level of performance of the hardware. You can assume that you get at least this amount of performance in any software.
This is much more telling than some figure that may be reached in software that has undergone the same level of extreme (and yes, I find a difference of 1000 points extreme) 'optimizations', but tells nothing about how badly the lesser or non-optimized software will run, while this is by far the largest group.
To react on some of your quotes:
"Futuremark seems to have a whim about something and all of sudden the scores change drastically without any true explanation."
As I have said before, the scores only change on the products of one specific vendor. I think that the vendor should be doing the explaining here, not FM.
But the explanation they gave you was a rather clever and manipulative one. And you fell for it.
"I am supposed to read a press release and go, "Oh, so now after three quarters, 3DMark2003 is finally doing what it is supposed to." No specifics mentioned, no reason given, just "we made it objective now...""
No you aren't, as I already said, you should have contacted FM aswell, and reflect their side of the story in the article aswell. I believe that FM was quite clear in the PDF documents they released regarding 3dmark03 and driver guidelines over the past months anyway.
"I would like nothing more to see a benchmark from Futuremark that was great and would unite the world of hardware geeks. Futuremark has tremendous reach, an incredible brand penetration, and the ability to give their products to the end users for free. That simply is an incredible thing. Too bad their current benchmark is junk."
I think the benchmark is doing a fine job (especially if you look at all the separate test-results that it generates, not just the overall score).
I also think that articles such as yours are in the way of the unity of hardware geeks.
I also think that NVIDIA has a larger reach and brand penetration than Futuremark, which makes their marketing quite dangerous to Futuremark. NVIDIA has been actively trying to destroy the credibility of Futuremark since the first ATi card beat NVIDIA's offerings in their benchmarks. And gullible (I hate to say it, but it's the truth) 'journalists' such as yourself have been willing pawns in their hands.
Finally, I think their benchmark is not junk at all... I am a Direct3D developer myself, and I generally find the same trends in performance of my products on various hardware as their benchmarks indicate (obviously because my products have not received the special NVIDIA 'compiler optimizations' yet, just like Half Life 2 and Tomb Raider: Angel of Darkness indicated aswell, at first).
If I were to design and build a Direct3D benchmark, I have no idea what I should be doing differently from Futuremark, actually. The things they do make a lot of sense to developers like me, generally.
Perhaps you should study the differences between NVIDIA's hardware and the other DX9 offerings more closely, and figure out why NVIDIA really NEEDS their 'optimizations' to keep up with the competition.
A good start is this article: http://www.3dcenter.org/artikel/cinefx/index_e.php
There are also many discussions on the excellent forum of Beyond3d.
Also, read your own thoughts in the article...
"I do not much believe anything that I am told by NVIDIA anymore. NVIDIA has lost their credibility this year and that is not something that is easy to regain. They need to stop “PRing†us to death"
This is exactly what I said aswell, why did I say it? Because you don't practice what you preach! Just look at the assumptions you've made!
"Aside from all the cheating and optimization, 3DMark2003 is simply too narrow of a look at gaming to accurately represent the big picture."
You'd be surprised how accurate 3DMark03 really is, in a variety of scenarios.
Performance all comes down to a few things: "How many texels can I fetch and how many instructions can I process in a given time?".
Since 3DMark03 has various shader and texture tests, and also has some complete scenes with the latest technologies applied (such as dynamic shadows, skinning, per-pixel lighting with normalmaps, Depth-of-Field, to name but a few..), it paints a very detailed picture of the performance of DX9-hardware. It also did this long before any DX9-games were available, and virtually all DX9-games that have been released since, have verified the large gap in performance between ATi and NVIDIA that 3DMark03 has shown since the beginning (before all the cheating started).
The bottom line is this: NVIDIA has taken a different approach to graphics than its competitors, and this places them at a disadvantage in DX9-software. Futuremark had to pay heavily for being the first to show this problem, basically it had to pay for the fact that it was such a good benchmark, that brought out the truth about performance.
NVIDIA basically sabotaged the product by introducing cheats to try and close the gap in 3DMark03 (and who knows how many other titles. Don't you find it remarkable that ATi always performs well in every new DX9-title from day 1, while NVIDIA has to wait for the next driver update to 'solve' the performance-problems).
Why do you still insist that this is all Futuremark's fault, and that their product is faulty?
I agree that 3DMark03 is only useful when there is no big cheating going on in the drivers, but you cannot blame Futuremark for the cheating, after all, they are not the ones doing it! In fact, they are working hard to try and eliminate cheating, and maintain the integrity of the benchmark. Their effort should be applauded, and NVIDIA's efforts should be banished.
Why are you doing the opposite? And worse: why don't you inform yourself properly before making these judgements in public?
In the meantime I found two sites that DID give a proper report, and had FM's explanation on the patches...
Take a look at this:
http://www.theinquirer.net/?article=12641
http://www.xbitlabs.com/news/video/display/20031112181114.html
I hope you will actually do something about this, rather than just thinking you're above people like me and sites like this, and think you don't have to change anything anyway (it happens).
Even NVIDIA agrees with me now:
http://www.xbitlabs.com/news/video/display/20031114041519.html
The actual compiler is not disabled, only the shader replacer.
And clearly the shader replacer should not even be active during 3DMark03, and the updated shaders clearly lead to a failing shader replacer, which is not a deliberate action of Futuremark to disable some part of the NVIDIA compiler specifically, but is just a consequence of a failing cheat-mechanism due to the patch, exactly as Futuremark intended.
So, it seems that you should be asking NVIDIA for a new explanation...
I am looking forward to your updated article.
To conclude, I absolutely think you are right that the 'gaming experience' is the most important... But keep in mind that this is a highly subjective and personal thing. I hope you will make it clear that your reviews describe YOUR gaming experience. You may favour speed over quality, but others might not, for example.
And also make it clear that 'gaming experience' has two components: gaming and experience. Meaning that the experience depends on the game in question, thanks to the 'optimizations' done by various manufacturers.
I also hope that you will still give objective information, like framerates in games, or all results in 3DMark03 (the overall score is really not that interesting, the separate shader, fillrate and other tests tell a much more complete story). Because some people (such as myself) are still highly interested in this kind of information, and build their own opinions of the reviewed hardware based on this. I hope you will try to write reviews where there is enough information for us to still make this possible.
Keep up the good work,
Scali
======
Kyle never gave a proper response at all (Only a "Thanks for your thoughts.", which may aswell have been an automated response). So I decided to send the mail to Steve aswell.
He replied, but he tried to defend it all as just opinions... Basically it came down to these two lines:
"it is simply our opinion, on our site."
...
"I fully support your right to an opinion, just as we have a right to disagree with the current trend in benchmarks, you have a right to defend it. We can respectfully agree to disagree."
This is fine, except... Opinions shouldn't be in news-items, that is not proper journalism. News should be provided as objectively as posssible. There is a responsibility to your readers.
Besides, not everything was just opinion. There were also skewed facts or even lies.
I tried to point this out in my next mail:
And if this is just your opinion, something must have gone wrong, the article doesn't make it clear at all that it is just your opinion, it presents many things as truths. This is not proper journalism as I said, journalism is about trying to reflect all sides of a story as objectively as possible.
And on top of that, some of the things written there are simply lies or assumptions, without even informing with Futuremark...
And now again I read that you slag off Futuremark in the message about Microsoft. There is something severely wrong here...
You should be reporting the news, not adding your own subjective remarks...
Certainly not in the way you did...
It only shows that you do not understand anything about the whole Futuremark issue.
What did Futuremark say? They only allow approved drivers now. So this little remark:
“your system specs meet the system requirements of this game. To ensure your game will continue to be compatible, please run this diagnostic again tomorrow…we might change system ratings by thenâ€.
is totally wrong and out-of-place.
The system ratings never changed. The exact same routines are benchmarked, and the exact same formula is used to calculate the final score (and with any card other than NVIDIA, the results are identical with either version, please think and draw some smart conclusions). Only the way NVIDIA handled the benchmark with their drivers changed. Futuremark tried to disable the cheats once, but NVIDIA just kept going... Now they are disabled for a second time, but here is the difference: this time you can only submit scores to the database if you have a driver that Futuremark approved to be handling the benchmark correctly, so no cheating.
NVIDIA can cheat all they want now, but Futuremark will simply not accept the results in their database now, so they will not influence the Game advisor.
When will you realize that the Futuremark standard never changed? You said it again. If you can't see what's really happening here, even after my explanations and links to other sources of useful info, then I can draw no other conclusion than that you are not capable of understanding modern hardware and software technology, and you are not fit for your job. Please respect the responsibility to your readers!
Scali
======
The reply I got was this:
I could've sworn this was our site.
Feel to use your own site http://www.scali.eu.org/Main.html to post your own opinions, I will provide a link so people can come to your site and tell you how to run it and call you names.
======
Apparently these people don't care what their readers think, and don't care whether what they say is right or not.
After the replies got increasingly cocky, I decided to just give up, and told them I would publish the emails, "hoping that it will save at least some of the less experienced from falling for your lies, because apparently you feel no need to change anything about the garbage that is on the site."
So here you are. I hope that people will go to more trustworthy sites when they want to read about 3d graphics software and hardware.
Scali
