It was only a matter of time

[Doomtrooper]

http://www.futuremark.com/community/halloffame/

1. NVIDIA GeForceFX 5900 Ultra

2. ATI RADEON 9800 PRO

3. ATI RADEON 9800

4. ATI RADEON 9700 PRO

5. NVIDIA GeForceFX 5900

6. ATI RADEON 9700

7. ATI RADEON 9500

8. NVIDIA GeForce FX 5800 Ultra

9. ATI RADEON 9600 Series

10. NVIDIA GeForce FX 5800

ATI should remove themself from this useless benchmark...all IHVs in fact

 

[Slides]

3DMark03 is a relevant benchmark, or so everyone on this forum keeps telling me.

 

[Jerky]

There should be some sort of artifact testing built into 3DMark. At any sign of tearing etc, the benchmark would abort. It's too easy to overclock a card and get a "boosted" 3DMark while the image is ripping and tearing everywhere.

 

[Bouncing Zabaglione Bros.]

3DMark03 is a relevant benchmark, or so everyone on this forum keeps telling me.

It is if you understand it's context. It is designed to load your card as heavily as possible, much in the same way as a breaking strain test is designed to test the strength of a cable. In order for the test to be relevant, the card must:

1. Perform a requested workload.
2. Perform the workload in the manner requested.
3. Output the image as requested by the benchmark.

The problem with "relevance" is that Nvidia do not perform the above. Their cheats mean that the results cannot be used for for comparison when dealing with Nvidia cards. They do not perform the test as requested, so their results are useless.

To question the relevance of the test because one manufacturer is cheating is rather like finding that a runner has cheated on a race, and therefore questioning the relevance of anyone running at all. Of course, what happens in the real world is that the cheater is disqualified, while everyone else runs the race fairly.

Also, note that the reasons why you might consider 3DMark2003 irrelevant as a test also hold true for other tests, such as UT2K or Quake. In the same way that Nvidia cheats on 3DMark2003 in order to get irrelevant scores, so too do they cheat in other benchmarks. Should we also abandon these tests as "irrelevant" too? Or should we disqualify Nvidia scores as tainted, cheating results?

 

[Pete]

This is really FM's fault, if the latest FX drivers still cheat on 3DM03. The onus is on them moreso than anyone else to make sure the scores submitted are valid, especially since the main value of 3DM is as a comparative tool.

 

[Bouncing Zabaglione Bros.]

This is really FM's fault, if the latest FX drivers still cheat on 3DM03. The onus is on them moreso than anyone else to make sure the scores submitted are valid, especially since the main value of 3DM is as a comparative tool.

Agreed. If the new NVdia drivers have simple reinstated the cheating, they should be yanked from the ORB. Is this another part of the Nvidia/Futuremark deal? Futuremark will ignore any cheats, and Nvidia will keep on cheating.

 

[Slides]

So, the consensus is that 3DMark03 is now irrelevant since they allow Nvidia to cheat?

 

[Myrmecophagavir]

I just wish they wouldn't publish these bollocks statistics. How come they can distinguish between 9700 and 9700 Pro, but not 9500 and 9500 Pro, or 9600 and 9600 Pro?

Top Performing DirectX8 GPUs
1. NVIDIA GeForce4 Ti 4600
2. NVIDIA GeForce4 Ti 4400
3. ATI RADEON 8500 Series
4. NVIDIA GeForce4 Ti 4200
5. ATI RADEON 9100 Series
6. ATI RADEON 9000
7. NVIDIA GeForce3 Ti 500
8. ATI MOBILITY RADEON 9000
9. NVIDIA GeForce3
10. NVIDIA GeForce3 Ti 200

How can the 9100 and 8500 come in at separate points, they are the same thing aren't they?

Top Performing DirectX7 GPUs
1. NVIDIA GeForce4 440 Go
2. NVIDIA nForce2
3. NVIDIA GeForce4 MX 460
4. NVIDIA GeForce4 MX 440
5. NVIDIA GeForce2 GTS/Pro/Ti
6. NVIDIA GeForce DDR
7. NVIDIA GeForce4 MX 420
8. ATI RADEON 7500 Series
9. NVIDIA GeForce2 Ultra
10. NVIDIA GeForce2 MX/MX 400

What's that GF2 Ultra doing down there? I'm surprised about the nForce2, Go 440 and 7500 results too but perhaps I'm just mis-remembering those products... aren't these stats vetted at all before being posted publicly?

 

[Bouncing Zabaglione Bros.]

So, the consensus is that 3DMark03 is now irrelevant since they allow Nvidia to cheat?

Not from me. As far as I am concerned, Nvidia scores are irrelevant as you cannot bench their cards accurately. 3Dmark2003 is no more or less broken than any other benchmark program, though its focus is different from most other benchmarking tools.

It's the faulty Nvidia drivers that do not run the test as the application requests that are broken. If it was a game that was getting reduced image quality, or having static clip planes inserted, we'd be calling the drivers buggy. Oh that's right, faulty filtering actually happens in UT2K.

However, to take my previous analogy, Nvidia is the cheat, but in order for the whole process to be considered fair and valid, then the referee (Futuremark) should do the "right thing" when seeing this cheat, and kick the Nvidia scores out of the ORB.

If you cheat in a race, you don't get to keep claiming to be the winner.

 

[breez]

Top Performing DirectX8 GPUs
1. NVIDIA GeForce4 Ti 4600
2. NVIDIA GeForce4 Ti 4400
3. ATI RADEON 8500 Series
4. NVIDIA GeForce4 Ti 4200
5. ATI RADEON 9100 Series
6. ATI RADEON 9000
7. NVIDIA GeForce3 Ti 500
8. ATI MOBILITY RADEON 9000
9. NVIDIA GeForce3
10. NVIDIA GeForce3 Ti 200

How can the 9100 and 8500 come in at separate points, they are the same thing aren't they?

The clockspeeds are different and Radeon 8500/LE overclocks better. A stock Radeon 8500 is 275/275 and Radeon 8500LE should be 250/250 (with some 3rd party manufacturers, there are lowerclocked versions too). 9100 is 250/250 at the best and only for one manufacturer's one specific model AFAIK (Club3D 128MB model). For example Club3D 64MB model is 250/183 and Sapphire 128MB model is 250/200

 

[Doomtrooper]

http://www.nvnews.net/vbulletin/showthread.php?s=8123c9bcd4720a29e106a4ddd19237af&threadid=15684

 

[Slides]

So, Nvidia video cards/drivers results are irrelevant since they cheat? Why not stop reviewing them if Nvidia is known to cheat? Is it not unfair to Ati to keep comparing them with a product that is known to be cheating?

 

[mczak]

Well the 9100 and 8500 can be in different spots since the 8500 includes both normal and LE cards, whereas the 9100 is always the slower version, except of course for those cards which have been upgraded with a bios flash (and sometimes slower a lot, some of the 9100 are clocked 250/230, some 250/200 and some have only 64bit memory busses).

Top Performing DirectX7 GPUs
1. NVIDIA GeForce4 440 Go
2. NVIDIA nForce2
3. NVIDIA GeForce4 MX 460
4. NVIDIA GeForce4 MX 440
5. NVIDIA GeForce2 GTS/Pro/Ti
6. NVIDIA GeForce DDR
7. NVIDIA GeForce4 MX 420
8. ATI RADEON 7500 Series
9. NVIDIA GeForce2 Ultra
10. NVIDIA GeForce2 MX/MX 400

What's that GF2 Ultra doing down there? I'm surprised about the nForce2, Go 440 and 7500 results too but perhaps I'm just mis-remembering those products... aren't these stats vetted at all before being posted publicly?

This chart is really hosed. Probably because there are insufficient number of samples. Futuremark says this list is based on build 330, and 2Ghz cpus. For example, if you look for the geforce2 ultra in the orb (all cpus, all cpu frequencies, build 330), the orb will there you there are exactly zero entries. Granted, you can see only the public entries, not all that were submitted, so I'd guess there is probably at least one entry (with a hosed configuration, unfortunately).
But I'm really not sure why futuremark bothers to publish that, since they have said numerous times that 3dmark03 is not a benchmark to evaluate performance of DX7 (and "early DX8") chips. That the results obviously do not reflect the performance you'd get with properly configured systems doesn't help exactly either.

 

[Bouncing Zabaglione Bros.]

So, Nvidia video cards/drivers results are irrelevant since they cheat? Why not stop reviewing them if Nvidia is known to cheat? Is it not unfair to Ati to keep comparing them with a product that is known to be cheating?

Yes, correct. That is why there is such as backlash growing against those reviewers/sites who either do not care, or do not have the technical understanding to see what is going on and inform their readers.

It's one of the reasons that ORB yanked all Nvidia scores with particular drivers before the 330 "anti-cheat" patch, but since then there have been political climbdowns from Futuremark when faced with Nvidia's "big stick" of lawyers. This has damaged their reputation, but the benchmarking tool itself hasn't changed and still is useful *if* you understand what it is measuring, the context of those results, and those products/companies that should be disregarded (by you the customer) because they won't play fair and follow the three rules I mentioned right at the top of this thread.

There are loads of threads about this all over the board if you want to search back through the last couple of months for more gory details.

 

[Unit01]

FM should pull and dis-allow all FX series card results with all driver revisions with the line "due to uncertainty in valid results"
Cause it's perfectly true. And seeing the CheatoFX up there isn't very good for FM's reputation for those that still have some faith in them :?

 

[Atomahawk]

I for one take these numbers with a serious grain of salt, and think that FM should reconsider even publishing such numbers without a thorough verification of certain things that may or may not actually effect the benchmark. 1) Issue of benchmark validity versus driver version should be top on their list. As they created the program, they should be held responsible and accountable to make certain the drivers are clean of any benchmark manipulation. 2) The drivers in question should be only WHQL certified drivers published by the card maker on their web sight. This is common sense considering certain drivers may not be meant to be published in the first place and may have enhancements not intended for released version (this protects FM and the card maker from any misunderstandings). 3) The issue of artifacts during benchmarking should be seriously considered. If FM considers their product to be forward looking as to possible DX9 performance in a game, then a threshold should be implemented as to the amount of actually artifacts that the benchmark should tolerate. I don’t know if they can measure this in the program, but it would certainly be a welcomed feature. What’s the point of a number if at the number a game is unplayable?
I’m certain that we can add more to this short list but I think this is the bare minimum.

FM is making a living with their program, companies pay to be in it, the burden lye’s with them as to the validity of these published numbers. If they choose to ignore this fact, well it obviously says a lot about their motives for the program.

This is not a criticism of FM as I believe their intentions are honest, but they can’t have it both ways, either they run a tight ship or they don’t. If they don’t it will be to their future demise.

Publishing those numbers without following stringent criteria puts them in a situation that I cannot understand, considering what they are all about.

 

[micron]

FM should pull and dis-allow all FX series card results with all driver revisions with the line "due to uncertainty in valid results"
Cause it's perfectly true. And seeing the CheatoFX up there isn't very good for FM's reputation for those that still have some faith in them :?

You dont seem to get it. How great of a benchmark could it be if it only let ATi cards submit scores? Do you think that it would be healthy for them financially to keep this up? Futuremark wants all IHV's to be able to participate, and they're going to make it happen....somehow..

 

[Bouncing Zabaglione Bros.]

You dont seem to get it. How great of a benchmark could it be if it only let ATi cards submit scores? Do you think that it would be healthy for them financially to keep this up? Futuremark wants all IHV's to be able to participate, and they're going to make it happen....somehow..

How great of a benchmark will it be if Futuremark allow people to cheat the scores and end up with cards that bench enormous numbers, but that give decidely sub-average performance when actually playing games?

It's the very core of gathering these statistics that the data must be a measurement of what the test intends to measure, not whatever the driver team decides to give you in place of running the test.

If the Nvidia cards are not running the full workload, in the manner requested, and with the output as required, they need to be disqualified.

 

[micron]

You dont seem to get it. How great of a benchmark could it be if it only let ATi cards submit scores? Do you think that it would be healthy for them financially to keep this up? Futuremark wants all IHV's to be able to participate, and they're going to make it happen....somehow..

How great of a benchmark will it be if Futuremark allow people to cheat the scores and end up with cards that bench enormous numbers, but that give decidely sub-average performance when actually playing games?

It's the very core of gathering these statistics that the data must be a measurement of what the test intends to measure, not whatever the driver team decides to give you in place of running the test.

If the Nvidia cards are not running the full workload, in the manner requested, and with the output as required, they need to be disqualified.

I totally agree with every word you just said
Futuremark though is in a tough position. They need to let Nvidia scores on the ORB somehow. Letting the FX scores be entered in is pissing alot of people off, but I can understand why their doing it. I'm just trying to put myself in Futuremarks shoes....

 

[Bouncing Zabaglione Bros.]

I totally agree with every word you just said
Futuremark though is in a tough position. They need to let Nvidia scores on the ORB somehow. Letting the FX scores be entered in is pissing alot of people off, but I can understand why their doing it. I'm just trying to put myself in Futuremarks shoes....

It's a short-term strategy. Once you are not policing the benchmark and there are doubts about the validity of the data, all the results are tainted by association in the eyes of the public. If Futuremark lose trust and objectivity, then that's just a slower way of killing the benchmark. They've already lost a lot of credibility by caving in to Nvidia. How do you think the other Futuremark members feel about that? What if ATI implement cheats? What if Dell doesn't want to be assiciated with such dishonesty?

To an extent, Futuremark have painted themselves into a corner, but if they don't stand up and do something *now* about cheating Nvidia scores, they are dead meat as far as the credibility of the ORB goes.