I think the real lesson today is that an alliance of reviewers and community volunteers should put together benchmark suites based on real world applications. These suites, which should only form one part of a reviewer's battary of benchmarks, should employ mechanisms to foil cheating. The source code of these suites should be kept confidential before reviews and released as open source software afterwards.
Otherwise, you have the situation today, of which HardOCP has made a reasonable assessment. Here is my brief assessment of the situation.
The flaws that were uncovered could point to several alternative scenarios:
1. High level management at Nvidia sanctioned cheating on 3DMark in the manner suspected.
2. Lower level employee at Nvidia cheated on the benchmark without management's knowledge.
3. Driver bug that was not detected by NVidia QC.
4. Driver bug that was detected but passed along in the hopes that people wouldn't notice.
5. High level management at 3DMark detected Nvidia cards and purposely introduced errors to descredit Nvidia.
6. Lower level employees of 3DMark did what is described in item 6.
7. Bug in 3DMark that only affects Nvidia hardware or software.
I am sure there are other scenarios that I am not covering. Is evidence present to support any one of these scenarios over the other ones? Without open source code produced by an independent organization with very limited commercial interests, how would one ever know?
Otherwise, you have the situation today, of which HardOCP has made a reasonable assessment. Here is my brief assessment of the situation.
The flaws that were uncovered could point to several alternative scenarios:
1. High level management at Nvidia sanctioned cheating on 3DMark in the manner suspected.
2. Lower level employee at Nvidia cheated on the benchmark without management's knowledge.
3. Driver bug that was not detected by NVidia QC.
4. Driver bug that was detected but passed along in the hopes that people wouldn't notice.
5. High level management at 3DMark detected Nvidia cards and purposely introduced errors to descredit Nvidia.
6. Lower level employees of 3DMark did what is described in item 6.
7. Bug in 3DMark that only affects Nvidia hardware or software.
I am sure there are other scenarios that I am not covering. Is evidence present to support any one of these scenarios over the other ones? Without open source code produced by an independent organization with very limited commercial interests, how would one ever know?
