View Full Version : It was only a matter of time
Doomtrooper
30-Jul-2003, 18:37
http://www.futuremark.com/community/halloffame/
1. NVIDIA GeForceFX 5900 Ultra
2. ATI RADEON 9800 PRO
3. ATI RADEON 9800
4. ATI RADEON 9700 PRO
5. NVIDIA GeForceFX 5900
6. ATI RADEON 9700
7. ATI RADEON 9500
8. NVIDIA GeForce FX 5800 Ultra
9. ATI RADEON 9600 Series
10. NVIDIA GeForce FX 5800
http://www.computerpannen.com/cwm/cwm/cwm/eek7.gif
http://common.ziffdavisinternet.com/util_get_image/2/0,3363,sz=1&i=24047,00.jpg
ATI should remove themself from this useless benchmark...all IHVs in fact :!:
3DMark03 is a relevant benchmark, or so everyone on this forum keeps telling me.
There should be some sort of artifact testing built into 3DMark. At any sign of tearing etc, the benchmark would abort. It's too easy to overclock a card and get a "boosted" 3DMark while the image is ripping and tearing everywhere.
Bouncing Zabaglione Bros.
30-Jul-2003, 19:21
3DMark03 is a relevant benchmark, or so everyone on this forum keeps telling me.
It is if you understand it's context. It is designed to load your card as heavily as possible, much in the same way as a breaking strain test is designed to test the strength of a cable. In order for the test to be relevant, the card must:
1. Perform a requested workload.
2. Perform the workload in the manner requested.
3. Output the image as requested by the benchmark.
The problem with "relevance" is that Nvidia do not perform the above. Their cheats mean that the results cannot be used for for comparison when dealing with Nvidia cards. They do not perform the test as requested, so their results are useless.
To question the relevance of the test because one manufacturer is cheating is rather like finding that a runner has cheated on a race, and therefore questioning the relevance of anyone running at all. Of course, what happens in the real world is that the cheater is disqualified, while everyone else runs the race fairly.
Also, note that the reasons why you might consider 3DMark2003 irrelevant as a test also hold true for other tests, such as UT2K or Quake. In the same way that Nvidia cheats on 3DMark2003 in order to get irrelevant scores, so too do they cheat in other benchmarks. Should we also abandon these tests as "irrelevant" too? Or should we disqualify Nvidia scores as tainted, cheating results?
This is really FM's fault, if the latest FX drivers still cheat on 3DM03. The onus is on them moreso than anyone else to make sure the scores submitted are valid, especially since the main value of 3DM is as a comparative tool.
Bouncing Zabaglione Bros.
30-Jul-2003, 20:14
This is really FM's fault, if the latest FX drivers still cheat on 3DM03. The onus is on them moreso than anyone else to make sure the scores submitted are valid, especially since the main value of 3DM is as a comparative tool.
Agreed. If the new NVdia drivers have simple reinstated the cheating, they should be yanked from the ORB. Is this another part of the Nvidia/Futuremark deal? Futuremark will ignore any cheats, and Nvidia will keep on cheating.
So, the consensus is that 3DMark03 is now irrelevant since they allow Nvidia to cheat?
Myrmecophagavir
30-Jul-2003, 23:51
I just wish they wouldn't publish these bollocks statistics. How come they can distinguish between 9700 and 9700 Pro, but not 9500 and 9500 Pro, or 9600 and 9600 Pro?
Top Performing DirectX8 GPUs
1. NVIDIA GeForce4 Ti 4600
2. NVIDIA GeForce4 Ti 4400
3. ATI RADEON 8500 Series
4. NVIDIA GeForce4 Ti 4200
5. ATI RADEON 9100 Series
6. ATI RADEON 9000
7. NVIDIA GeForce3 Ti 500
8. ATI MOBILITY RADEON 9000
9. NVIDIA GeForce3
10. NVIDIA GeForce3 Ti 200
How can the 9100 and 8500 come in at separate points, they are the same thing aren't they?
Top Performing DirectX7 GPUs
1. NVIDIA GeForce4 440 Go
2. NVIDIA nForce2
3. NVIDIA GeForce4 MX 460
4. NVIDIA GeForce4 MX 440
5. NVIDIA GeForce2 GTS/Pro/Ti
6. NVIDIA GeForce DDR
7. NVIDIA GeForce4 MX 420
8. ATI RADEON 7500 Series
9. NVIDIA GeForce2 Ultra
10. NVIDIA GeForce2 MX/MX 400
What's that GF2 Ultra doing down there? I'm surprised about the nForce2, Go 440 and 7500 results too but perhaps I'm just mis-remembering those products... aren't these stats vetted at all before being posted publicly?
Bouncing Zabaglione Bros.
31-Jul-2003, 00:20
So, the consensus is that 3DMark03 is now irrelevant since they allow Nvidia to cheat?
Not from me. As far as I am concerned, Nvidia scores are irrelevant as you cannot bench their cards accurately. 3Dmark2003 is no more or less broken than any other benchmark program, though its focus is different from most other benchmarking tools.
It's the faulty Nvidia drivers that do not run the test as the application requests that are broken. If it was a game that was getting reduced image quality, or having static clip planes inserted, we'd be calling the drivers buggy. Oh that's right, faulty filtering actually happens in UT2K.
However, to take my previous analogy, Nvidia is the cheat, but in order for the whole process to be considered fair and valid, then the referee (Futuremark) should do the "right thing" when seeing this cheat, and kick the Nvidia scores out of the ORB.
If you cheat in a race, you don't get to keep claiming to be the winner.
Top Performing DirectX8 GPUs
1. NVIDIA GeForce4 Ti 4600
2. NVIDIA GeForce4 Ti 4400
3. ATI RADEON 8500 Series
4. NVIDIA GeForce4 Ti 4200
5. ATI RADEON 9100 Series
6. ATI RADEON 9000
7. NVIDIA GeForce3 Ti 500
8. ATI MOBILITY RADEON 9000
9. NVIDIA GeForce3
10. NVIDIA GeForce3 Ti 200
How can the 9100 and 8500 come in at separate points, they are the same thing aren't they?
The clockspeeds are different and Radeon 8500/LE overclocks better. A stock Radeon 8500 is 275/275 and Radeon 8500LE should be 250/250 (with some 3rd party manufacturers, there are lowerclocked versions too). 9100 is 250/250 at the best and only for one manufacturer's one specific model AFAIK (Club3D 128MB model). For example Club3D 64MB model is 250/183 and Sapphire 128MB model is 250/200 :lol:
Doomtrooper
31-Jul-2003, 00:54
http://www.nvnews.net/vbulletin/showthread.php?s=8123c9bcd4720a29e106a4ddd19237af& threadid=15684
:lol:
So, Nvidia video cards/drivers results are irrelevant since they cheat? Why not stop reviewing them if Nvidia is known to cheat? Is it not unfair to Ati to keep comparing them with a product that is known to be cheating?
Well the 9100 and 8500 can be in different spots since the 8500 includes both normal and LE cards, whereas the 9100 is always the slower version, except of course for those cards which have been upgraded with a bios flash (and sometimes slower a lot, some of the 9100 are clocked 250/230, some 250/200 and some have only 64bit memory busses).
Top Performing DirectX7 GPUs
1. NVIDIA GeForce4 440 Go
2. NVIDIA nForce2
3. NVIDIA GeForce4 MX 460
4. NVIDIA GeForce4 MX 440
5. NVIDIA GeForce2 GTS/Pro/Ti
6. NVIDIA GeForce DDR
7. NVIDIA GeForce4 MX 420
8. ATI RADEON 7500 Series
9. NVIDIA GeForce2 Ultra
10. NVIDIA GeForce2 MX/MX 400
What's that GF2 Ultra doing down there? I'm surprised about the nForce2, Go 440 and 7500 results too but perhaps I'm just mis-remembering those products... aren't these stats vetted at all before being posted publicly?
This chart is really hosed. Probably because there are insufficient number of samples. Futuremark says this list is based on build 330, and 2Ghz cpus. For example, if you look for the geforce2 ultra in the orb (all cpus, all cpu frequencies, build 330), the orb will there you there are exactly zero entries. Granted, you can see only the public entries, not all that were submitted, so I'd guess there is probably at least one entry (with a hosed configuration, unfortunately).
But I'm really not sure why futuremark bothers to publish that, since they have said numerous times that 3dmark03 is not a benchmark to evaluate performance of DX7 (and "early DX8") chips. That the results obviously do not reflect the performance you'd get with properly configured systems doesn't help exactly either.
Bouncing Zabaglione Bros.
31-Jul-2003, 07:48
So, Nvidia video cards/drivers results are irrelevant since they cheat? Why not stop reviewing them if Nvidia is known to cheat? Is it not unfair to Ati to keep comparing them with a product that is known to be cheating?
Yes, correct. That is why there is such as backlash growing against those reviewers/sites who either do not care, or do not have the technical understanding to see what is going on and inform their readers.
It's one of the reasons that ORB yanked all Nvidia scores with particular drivers before the 330 "anti-cheat" patch, but since then there have been political climbdowns from Futuremark when faced with Nvidia's "big stick" of lawyers. This has damaged their reputation, but the benchmarking tool itself hasn't changed and still is useful *if* you understand what it is measuring, the context of those results, and those products/companies that should be disregarded (by you the customer) because they won't play fair and follow the three rules I mentioned right at the top of this thread.
There are loads of threads about this all over the board if you want to search back through the last couple of months for more gory details.
FM should pull and dis-allow all FX series card results with all driver revisions with the line "due to uncertainty in valid results"
Cause it's perfectly true. And seeing the CheatoFX up there isn't very good for FM's reputation for those that still have some faith in them :?
Atomahawk
31-Jul-2003, 15:23
I for one take these numbers with a serious grain of salt, and think that FM should reconsider even publishing such numbers without a thorough verification of certain things that may or may not actually effect the benchmark. 1) Issue of benchmark validity versus driver version should be top on their list. As they created the program, they should be held responsible and accountable to make certain the drivers are clean of any benchmark manipulation. 2) The drivers in question should be only WHQL certified drivers published by the card maker on their web sight. This is common sense considering certain drivers may not be meant to be published in the first place and may have enhancements not intended for released version (this protects FM and the card maker from any misunderstandings). 3) The issue of artifacts during benchmarking should be seriously considered. If FM considers their product to be forward looking as to possible DX9 performance in a game, then a threshold should be implemented as to the amount of actually artifacts that the benchmark should tolerate. I don’t know if they can measure this in the program, but it would certainly be a welcomed feature. What’s the point of a number if at the number a game is unplayable?
I’m certain that we can add more to this short list but I think this is the bare minimum.
FM is making a living with their program, companies pay to be in it, the burden lye’s with them as to the validity of these published numbers. If they choose to ignore this fact, well it obviously says a lot about their motives for the program.
This is not a criticism of FM as I believe their intentions are honest, but they can’t have it both ways, either they run a tight ship or they don’t. If they don’t it will be to their future demise.
Publishing those numbers without following stringent criteria puts them in a situation that I cannot understand, considering what they are all about.
FM should pull and dis-allow all FX series card results with all driver revisions with the line "due to uncertainty in valid results"
Cause it's perfectly true. And seeing the CheatoFX up there isn't very good for FM's reputation for those that still have some faith in them :?You dont seem to get it. How great of a benchmark could it be if it only let ATi cards submit scores? Do you think that it would be healthy for them financially to keep this up? Futuremark wants all IHV's to be able to participate, and they're going to make it happen....somehow..
Bouncing Zabaglione Bros.
31-Jul-2003, 18:28
You dont seem to get it. How great of a benchmark could it be if it only let ATi cards submit scores? Do you think that it would be healthy for them financially to keep this up? Futuremark wants all IHV's to be able to participate, and they're going to make it happen....somehow..
How great of a benchmark will it be if Futuremark allow people to cheat the scores and end up with cards that bench enormous numbers, but that give decidely sub-average performance when actually playing games?
It's the very core of gathering these statistics that the data must be a measurement of what the test intends to measure, not whatever the driver team decides to give you in place of running the test.
If the Nvidia cards are not running the full workload, in the manner requested, and with the output as required, they need to be disqualified.
You dont seem to get it. How great of a benchmark could it be if it only let ATi cards submit scores? Do you think that it would be healthy for them financially to keep this up? Futuremark wants all IHV's to be able to participate, and they're going to make it happen....somehow..
How great of a benchmark will it be if Futuremark allow people to cheat the scores and end up with cards that bench enormous numbers, but that give decidely sub-average performance when actually playing games?
It's the very core of gathering these statistics that the data must be a measurement of what the test intends to measure, not whatever the driver team decides to give you in place of running the test.
If the Nvidia cards are not running the full workload, in the manner requested, and with the output as required, they need to be disqualified.
I totally agree with every word you just said :wink:
Futuremark though is in a tough position. They need to let Nvidia scores on the ORB somehow. Letting the FX scores be entered in is pissing alot of people off, but I can understand why their doing it. I'm just trying to put myself in Futuremarks shoes....
Bouncing Zabaglione Bros.
31-Jul-2003, 20:23
I totally agree with every word you just said :wink:
Futuremark though is in a tough position. They need to let Nvidia scores on the ORB somehow. Letting the FX scores be entered in is pissing alot of people off, but I can understand why their doing it. I'm just trying to put myself in Futuremarks shoes....
It's a short-term strategy. Once you are not policing the benchmark and there are doubts about the validity of the data, all the results are tainted by association in the eyes of the public. If Futuremark lose trust and objectivity, then that's just a slower way of killing the benchmark. They've already lost a lot of credibility by caving in to Nvidia. How do you think the other Futuremark members feel about that? What if ATI implement cheats? What if Dell doesn't want to be assiciated with such dishonesty?
To an extent, Futuremark have painted themselves into a corner, but if they don't stand up and do something *now* about cheating Nvidia scores, they are dead meat as far as the credibility of the ORB goes.
What would you have them do?.....run it by me please......
Bouncing Zabaglione Bros.
31-Jul-2003, 21:38
What would you have them do?.....run it by me please......
Check the drivers the scores were taken with. If they contain cheats that invalidate the results, yank the scores from the ORB. The same should go for any manufacturer. It's the equivalent of not counting a spoiled ballot vote or a spoiled test score paper.
Myrmecophagavir
31-Jul-2003, 23:09
Isn't it the case that newer Detonator revisions are WHQL only for the FX series and not for the rest of the cards? Does that mean you can upload scores from FX cards using those drivers but not from other cards, or does the ORB just check the version number and not which card it was used with?
Bouncing Zabaglione Bros.
31-Jul-2003, 23:21
Isn't it the case that newer Detonator revisions are WHQL only for the FX series and not for the rest of the cards? Does that mean you can upload scores from FX cards using those drivers but not from other cards, or does the ORB just check the version number and not which card it was used with?
Public WHQL drivers are a requirement of the 3Dmark2003 licence, but WHQL doesn't stop any drivers that identify applications and then implement cheats.
I suppose the WHQL requirement was a way of making the scores repeatable, and to prevent any special developer drivers with the sole target of racking up huge ORB scores for marketing purposes. I don't think Futuremark expected such a massive effort from Nvidia to write and hide such cheats, even going to the extent of encrypting their drivers.
What would you have them do?.....run it by me please......
Check the drivers the scores were taken with. If they contain cheats that invalidate the results, yank the scores from the ORB. The same should go for any manufacturer. It's the equivalent of not counting a spoiled ballot vote or a spoiled test score paper.
Do you happen to know which drivers the 5900Ultra sitting in the ORB's top position was using?
In order for that card to grab the top spot and get published, the drivers had to have been aproved by FM in the first place. Futuremark is saying that it is ok for that card to hold the number one position.
We all know, including FM, that Nvidia is going to cheat in every driver release, but FM cannot survive without letting FX cards submit scores.
Bouncing Zabaglione Bros.
01-Aug-2003, 01:44
Do you happen to know which drivers the 5900Ultra sitting in the ORB's top position was using?
In order for that card to grab the top spot and get published, the drivers had to have been aproved by FM in the first place. Futuremark is saying that it is ok for that card to hold the number one position.
We all know, including FM, that Nvidia is going to cheat in every driver release, but FM cannot survive without letting FX cards submit scores.
No, sorry I don't know. However, IIRC soon after the 330 patch came out and dropped the GFFX scores, another Nvidia driver came out that put the scores back up to cheat levels. We've had no comment on that from Futuremark one way or the other.
Given what we know of the Futuremark climbdown, and the other Nvidia cheats still in place, I can only infer that GFFX could only have reached that score by cheating as per previous drivers. It's poor shader performance should have seen to that.
I'm not very happy about taking this "guilty until proven innocent" approach, but based on past performance and the lack of comment from Futuremark or Nvidia, I need to be convinced the cheats are not happening. Caveat emptor and all that. Nvidia has used up all their free passes.
As I said before, letting cheating drivers submit scores simply devalues the long term reputation of Futuremark and their products. What happens when ATI decides to do the same? The whole thing becomes useless.
CorwinB
01-Aug-2003, 06:03
The whole thing becomes useless.
The whole thing already is, actually... FutureMark is caught between not letting the FX line post scores (if the currently approved drivers on the ORB cheat, which may or may not be the case), which makes 3DM more or less useless if no comparison is possible, and letting the benchmark be cheated, which also makes comparisons meaningless.
And for most enthusiasts, Futuremark's reputation is already tarnished beyond repair : people are either turned against synthetic benchmarks, "thanks" to Nvidia's very efficient anti-3DMark smearing campaign using their "guys with webpages" as proxies, or don't trust FutureMark anymore because of the infamous "It's optimizations not cheats" FM/NV joint statement.
...
No, sorry I don't know. However, IIRC soon after the 330 patch came out and dropped the GFFX scores, another Nvidia driver came out that put the scores back up to cheat levels. We've had no comment on that from Futuremark one way or the other.
....
A problem I've had with looking at this issue 20-20 is that I've been spoiled rotten by ATi over the last year in getting an official driver release on average every 4-6 weeks. So, every time I hear something about the "latest" nVidia drivers I tend to think of them as company-sanctioned drivers which have been officially released. Just checked the nVidia site though and it looks like nVidia hasn't officially released a set of drivers since May 14, the 44.03's. Everything else apparently falls under the so-called "leaked" driver definition, and IMO, really doesn't count. That is, I wouldn't expect FM to release patches based on these "leaked" drivers. But I would and do expect them to release recompile patches every time nVidia releases an official driver, though. At least, if FM wants to preserve the tatters of its credibility, it should be prepared to do this automatically each time nVidia releases a new driver set officially. If they don't do this then IMO it will simply tell me that FM just no longer cares whether its benchmarks are manipulated. I'll support FM only so long as they defend their software--when they stop defending themselves so will I. But I really don't think it's fair to expect them to respond to anything other than officially released Detonators. I think they have a little more rope, yet.
I would prefer it if FM would cease posting scores from non-official driver sets, regardless of IHV. With reference to "leaked" drivers no one really knows "who" is doing the "leaking" or what the drivers actually entail--they could easily be older sets, a mixture of older sets, renamed, etc. With official Detonators coming 3-4 months apart as they sometimes do I can understand people with nVidia products becoming enthusiastic about such "leaks", but nonetheless websites should not be evaluating products with such drivers. It's issues like these that make me really glad I can view the nVidia predicament from a distance these days...;)
Bouncing Zabaglione Bros.
02-Aug-2003, 17:56
I would prefer it if FM would cease posting scores from non-official driver sets, regardless of IHV.
Agreed. I thought it was made pretty clear that the 3DMark2003 licence agreement *requires* a publicly released WHQL driver (I speculated on the reasons why in an earlier post). If scores are submitted to the ORB that do not comply to this, they should be yanked from the ORB.
There's no point in having a score in the ORB if it was made with a hacked driver that renders a black screen at 500 fps for a 30,000+ final score.
Can somebody clarify which NVIDIA drivers after the 44.03s are definitely WHQL? I was under the impression that there aren't any - so that will preclude all ORB entries, whether published or not, from being included in the data sample for the HoF. I have the funny feeling that another revision is though...
Also note that the positioning problem of certain cards due to their chip names is a driver/IHV problem and not FMs; once everybody gets that sorted out and enough entries are submitted to the ORB, that particular issue should disappear. For example, the 44.03 drivers recognise a NV31 rev1 as a 5600 (no "Ultra) but labels a rev2 as being a 5600 Ultra.
Agreed. I thought it was made pretty clear that the 3DMark2003 licence agreement *requires* a publicly released WHQL driver (I speculated on the reasons why in an earlier post). If scores are submitted to the ORB that do not comply to this, they should be yanked from the ORB.
Actually the licence doesn't include entries to the ORB and as far as I can tell, the licence for the 330 version doesn't say anything about WHQL drivers at all.
dan2097
02-Aug-2003, 20:30
Can somebody clarify which NVIDIA drivers after the 44.03s are definitely WHQL? I was under the impression that there aren't any - so that will preclude all ORB entries, whether published or not, from being included in the data sample for the HoF. I have the funny feeling that another revision is though...
From guru3d the only higher than 44.03 drivers I can see that are WHQL are the popular EVGA 44.67:
http://download.guru3d.com/pafiledb.php?action=file&id=640
EDIT: thus these can distort the results of the HOF asuming they still cheat.
That's odd - the 44.67 drivers that I've been using aren't WHQL.
Dave Baumann
03-Aug-2003, 01:27
Nick, there are board vendor variants that are WHQL. Yours may not be WHQL, but other boards may be bundled with them post the WHQL process. I think there is at least one other set thr AIB's are bundling that are WHQL, but aren't available as an official download from NVIDIA.
Nick, there are board vendor variants that are WHQL. Yours may not be WHQL, but other boards may be bundled with them post the WHQL process. I think there is at least one other set thr AIB's are bundling that are WHQL, but aren't available as an official download from NVIDIA.
Dave, do you have any ideas on why this is done? I mean, it seems to me that if nVidia's going to WHQL a set of drivers for one of its OEMs there should be no problem releasing a concurrent WHQL version of its reference drivers officially. I'm also assuming that eVGA, like most OEMs, is 100% dependent on nVidia to supply it with its drivers.
If memory serves, I've also seen some mention of "WHQL" drivers which are WHQL for nv3x only (and not for other cards supported in the driver set) or else WHQL only for the older cards and not for the nv3x series.
I can understand cards shipped with non-WHQL drivers prior to the drivers completing the WHQL process, but am not clear on why nVidia would supply reference-design board drivers which are WHQL'ed to an OEM but fail to make them available as official WHQL reference drivers from its web site. I'm wondering now if nVidia isn't mixing in elements from older WHQL driver sets along with some newer elements, and shipping drivers with a newer version number to OEMs as "new WHQL drivers."
I don't have the time to read the whole thread ( or at least not now ) , but just one quick comment...
The cheat causing the artifact shown in the screenshot posted in the original post of this thread IS gone in some leaked drivers already, and will also be gone in future official releases. The same is true for the "no-back-buffer-clearing" optimization.
All shading optimizations remain and will continue to exist in future driver releases, at least in the near future.
Uttar
Dave Baumann
04-Aug-2003, 16:47
The cheat causing the artifact shown in the screenshot posted in the original post of this thread IS gone
They did miss one though.
digitalwanderer
04-Aug-2003, 16:49
The cheat causing the artifact shown in the screenshot posted in the original post of this thread IS gone
They did miss one though.
Which one?
I don't have the time to read the whole thread ( or at least not now ) , but just one quick comment...
The cheat causing the artifact shown in the screenshot posted in the original post of this thread IS gone in some leaked drivers already, and will also be gone in future official releases. The same is true for the "no-back-buffer-clearing" optimization.
All shading optimizations remain and will continue to exist in future driver releases, at least in the near future.
Uttar
I don't recall that being proved, only that the artifact is no longer present... that might simply mean the cheat detects the camera going off-rail and deactivating... unless I missed someone clearing up this issue.
Back to June there are no 5900/5600 or 9800 on the 2003 hall fame list,when I asked worm,he said the results are based on 2001 benchmark,and 5900/9800 benchmark results is very few.
Just 2 months past things changed,oh,so many people now using DX9 cards? :D
Hanners
05-Aug-2003, 14:38
The cheat causing the artifact shown in the screenshot posted in the original post of this thread IS gone
They did miss one though.
Which one?
I think it's one in GT3 - The "Troll's Lair" test. It's another not clearing the back-buffer 'optimisation' if I remember correctly. It's more difficult to spot, but it's still there.... Am I right?
Myrmecophagavir
05-Aug-2003, 19:14
Back to June there are no 5900/5600 or 9800 on the 2003 hall fame list,when I asked worm,he said the results are based on 2001 benchmark,and 5900/9800 benchmark results is very few.
Just 2 months past things changed,oh,so many people now using DX9 cards? :D
Yes, that makes sense doesn't it? 2 months is a large proportion of the lifespan so far for the 9800 and others, so the number of people using them would have gone up a lot. In addition people weren't so interested in uploading 3DMark2001 scores to the ORB for those cards, people only care about 3DMark03. There was no 03 hall of fame list in June.
dan2097
05-Aug-2003, 21:30
I think it's one in GT3 - The "Troll's Lair" test. It's another not clearing the back-buffer 'optimisation' if I remember correctly. It's more difficult to spot, but it's still there.... Am I right?
I thoguht it was a gt2 clip planes, am I right :P ?
Doomtrooper
13-Aug-2003, 15:09
Strategic BETA Members:
http://www.futuremark.com/betaprogram/images/sbeta_logos.gif
"In our testing, all identified detection mechanisms stopped working when we altered the benchmark code just trivially and without changing any of the actual benchmark workload. With this altered benchmark, NVIDIA’s certain products had a performance drop of as much as 24.1% while competition’s products performance drop stayed within the margin of error of 3%. To our knowledge, all drivers with these detection mechanisms were published only after the launch of 3DMark03. According to industry’s terminology, this type of driver design is defined as ‘driver
cheats’.
Members of Futuremark’s BETA program first noticed how parts of the tests in 3DMark03 were rendered differently on different hardware. When testing NVIDIA hardware on 3DMark03 with socalled developer’s version’s free camera enabled, they noticed how some parts of tests were rendered strangely, and informed Futuremark of their findings. Futuremark investigated further and our findings show that certain NVIDIA drivers seem to detect when 3DMark03 is running and then replace the 3DMark03’s rendering requests with manually implemented alternative rendering operations. These alternative rendering operations reduce the amount of rendering work and thereby increase the obtained benchmark result."
And the NVIDIA perspective...
"Since NVIDIA is not part in the FutureMark beta program (a program which costs of hundreds of thousands of dollars to participate in) we do not get a chance to work with Futuremark on writing the shaders like we would with a real applications developer. We don't know what they did, but it looks like they have intentionally tried to create a scenario that makes our products look bad. This is obvious since our relative performance on games like Unreal Tournament 2003 and Doom 3 shows that the GeForce FX 5900 Ultra is by far the fastest graphics on the market today."
Classic ...and predictable http://smilies.xibase.com/lol2.gif
digitalwanderer
13-Aug-2003, 15:40
This is just a leetle too much for me this morning, 'scuse me while I go puke up a bit of bile before continuing. :evil:
Anyone ever notice that nVidia PR has a wonderful tendancy to really crap up a nice morning?
This is just a leetle too much for me this morning, 'scuse me while I go puke up a bit of bile before continuing. :evil:
Anyone ever notice that nVidia PR has a wonderful tendancy to really crap up a nice morning?
If a PR representative can affect your morning, then maybe it's time to visit the beach and get some sun. ;)
digitalwanderer
13-Aug-2003, 16:39
This is just a leetle too much for me this morning, 'scuse me while I go puke up a bit of bile before continuing. :evil:
Anyone ever notice that nVidia PR has a wonderful tendancy to really crap up a nice morning?
If a PR representative can affect your morning, then maybe it's time to visit the beach and get some sun. ;)
Don't EVEN get me started on the e-mail I found from Mr.Perez in me box this morning, it was the freaking icing on the cake. :roll:
Beach is out, it's heavy over-cast and me daughter apparently got up VERY early and had some 'fun' around the house before waking us up....I'm still in the "recovering the damage" phase.
Err,Worm said that they have removed the current Hall of Fame:
"Just a quick note to all of you that we have removed the current Hall of Fame. We are working on a better solution and it will be published as soon as it is ready. Stay tuned... "
hmm,Just wondering what better solutions means? :roll:
incurable
19-Aug-2003, 16:40
hmm,Just wondering what better solutions means? :roll:
More nVidia cards on top. :roll:
Did I actually write that? :oops:
cu
incurable
Joe DeFuria
19-Aug-2003, 21:19
hmm,Just wondering what better solutions means? :roll:
More nVidia cards on top. :roll:
Did I actually write that? :oops:
cu
incurable
You may think it's funny, but there might be some sort of real truth to that.
AFAIK There have been issues with certain chipsets reporting as "the same product" to the Database. In other words, a Radeon 9800 and a Radeon 9800 Pro might both be reporting and grouping themselves as "9800". Same with nVidia cards.
This might be a re-vamp to try and actually separate differnt SKUs of the same chip? So you can distinguish a GeForce 5900 Ultra from a 5900, or a ATI 9500 from a 9500 Pro, etc.
If someone at the top has multiple SKUs that are being lumped together, perhaps they made some noise to rectify the situation. ;)
Seriously, I have no idea if this is the reason for the update, and I alsp don't have any idea who would actually benefit more, ATI or nVidia, from a change....
Or maybe Matrox will take the no.1 spot...
Nick[FM]
20-Aug-2003, 08:25
AFAIK There have been issues with certain chipsets reporting as "the same product" to the Database. In other words, a Radeon 9800 and a Radeon 9800 Pro might both be reporting and grouping themselves as "9800". Same with nVidia cards.
This might be a re-vamp to try and actually separate differnt SKUs of the same chip? So you can distinguish a GeForce 5900 Ultra from a 5900, or a ATI 9500 from a 9500 Pro, etc.
That was already done in the "old" HoF. We already got the (for example) Radeon 9800 and Radeon 9800 Pro, 9500 series and 9700 series all separated etc.
Patric Ojala
20-Aug-2003, 09:24
Err,Worm said that they have removed the current Hall of Fame:
"Just a quick note to all of you that we have removed the current Hall of Fame. We are working on a better solution and it will be published as soon as it is ready. Stay tuned... "
hmm,Just wondering what better solutions means? :roll:
I'm not sure I know the whole truth, but at least one reason for maintenance was that the old top list was affected by a large amount of 3DMark results run on unofficial drivers. Not leaked beta drivers, wörm just corrected me :lol: , but unofficial drivers anyway. We also have our still ongoing effort to define what driver optimizations could be considered acceptable.
That old Top List was basically just some averages of received benchmark results, not some "Futuremark's official best buy guide" as some people saw it as. We'll see what we can do about this list, and hope the new list will be less ambiguous.
K.I.L.E.R
20-Aug-2003, 09:29
I'm looking forward to it Mr Ojala. :)
Err,Worm said that they have removed the current Hall of Fame:
"Just a quick note to all of you that we have removed the current Hall of Fame. We are working on a better solution and it will be published as soon as it is ready. Stay tuned... "
hmm,Just wondering what better solutions means? :roll:
I'm not sure I know the whole truth, but at least one reason for maintenance was that the old top list was affected by a large amount of 3DMark results run on unofficial drivers (leaked beta drivers and similar). We also have our still ongoing effort to define what driver optimizations could be considered acceptable.
That old Top List was basically just some averages of received benchmark results, not some "Futuremark's official best buy guide" as some people saw it as. We'll see what we can do about this list, and hope the new list will be less ambiguous.
PatrickL
20-Aug-2003, 11:30
[We also have our still ongoing effort to define what driver optimizations could be considered acceptable.
I think it 's the problem. Could you explain please why FM thinks any optimization is acceptable for a benchmark ?
I agree. NO OPTIMIZATIONS FOR A BENCHMARK should be allowed.
K.I.L.E.R
20-Aug-2003, 14:00
I agree. The only optimisations that should be done are API specific optimisations that are all round.
Nick[FM]
20-Aug-2003, 14:07
I agree. NO OPTIMIZATIONS FOR A BENCHMARK should be allowed.
Out of curiosity, what about all games with built in benchmarks?
Joe DeFuria
20-Aug-2003, 14:28
Once and for all, can Futuremark make a clear and concise statement or set of statements that defines:
1) What the purpose of 3DMark is supposed to be. For example
a) Representative of a guess on absolute performance with DX8 and DX9 games
b) A "stress test" for GPUs, not designed to be indicative of absolute performance levels, but an indiciation of relative GPU power in given circumstances....and what the circumstances are designed to represent.
2) Qualify exactly what 3DMark "score" represents in your view.
a) Measurement of which GPU is "faster"
b) Measurement of Which GPU is "better", (and what, qualitatively speaking, defines "better")
In my opinion, the real problem with 3Dmark, is what appears to me to be an inconsistent presentation from FutureMark of what exactly 3DMark is, how it's supposed to be used, and what the score represents.
I really have no clear indication at this time WHAT exactly the 3DMark score is trying to convey. And until FutureMark can make a CLEAR and CONCISE statement about what the PURPOSE of the benchmark is, it is near impossible to discuss on whether or not it meets those objectives, or what is and is not "acceptable" in terms of optimizations, etc.
And yes, I've read the initial white paper, and I THOUGHT I had understood the purpose...right up through the denouncing of nVidia cheats...but recent events leave me perplexed.
PatrickL
20-Aug-2003, 14:50
I don't think you should move the talk to games used as a benchmark after the UT2003 issue. Or do you suggest that the trilinear filtering was removed in Nidia drivers for general gaming purpose as a general optimization :)
Doomtrooper
20-Aug-2003, 15:37
]I agree. NO OPTIMIZATIONS FOR A BENCHMARK should be allowed.
Out of curiosity, what about all games with built in benchmarks?
Depends on what the 'optimization' is, lets take the ever popular UT 2003 benchmark which has obviousally got some 'special attention' latley. Disabling trilinear is not a 'optimization' as you can see the image quality loss.
http://www.beyond3d.com/forum/viewtopic.php?t=7228&start=0
Replacing shaders in a benchmark that does not change the visual output like replacing the water shader would be welcomed in a 'game' if performance was increased and if the output stays the same vs the orginal reference, but Nvidia has failed on all accounts.
FX
http://www.bit-tech.net/images/review/187/gt4fxlq.jpg
9700
http://www.bit-tech.net/images/review/187/gt4radlq.jpg
digitalwanderer
20-Aug-2003, 15:45
I've read the initial white paper, and I THOUGHT I had understood the purpose...right up through the denouncing of nVidia cheats...but recent events leave me perplexed.
Sums it all up rather well right there. :)
]We also have our still ongoing effort to define what driver optimizations could be considered acceptable.
I think it 's the problem. Could you explain please why FM thinks any optimization is acceptable for a benchmark ?
I agree. NO OPTIMIZATIONS FOR A BENCHMARK should be allowed.
Out of curiosity, what about all games with built in benchmarks?
Worm,
Just answer the damn question please. We're not talking about games here, we're talking about 3DMark. 3DMark is not a game, but solely a synthetic benchmark. Please, answer the question without sending your response through some kind of lawyer interpreter.
Tommy McClain
Reverend
20-Aug-2003, 17:09
Once and for all, can Futuremark make a clear and concise statement or set of statements that defines:
1) What the purpose of 3DMark is supposed to be. For example
a) Representative of a guess on absolute performance with DX8 and DX9 games
b) A "stress test" for GPUs, not designed to be indicative of absolute performance levels, but an indiciation of relative GPU power in given circumstances....and what the circumstances are designed to represent.
2) Qualify exactly what 3DMark "score" represents in your view.
a) Measurement of which GPU is "faster"
b) Measurement of Which GPU is "better", (and what, qualitatively speaking, defines "better")
In my opinion, the real problem with 3Dmark, is what appears to me to be an inconsistent presentation from FutureMark of what exactly 3DMark is, how it's supposed to be used, and what the score represents.
I really have no clear indication at this time WHAT exactly the 3DMark score is trying to convey. And until FutureMark can make a CLEAR and CONCISE statement about what the PURPOSE of the benchmark is, it is near impossible to discuss on whether or not it meets those objectives, or what is and is not "acceptable" in terms of optimizations, etc.
And yes, I've read the initial white paper, and I THOUGHT I had understood the purpose...right up through the denouncing of nVidia cheats...but recent events leave me perplexed.
Joe, remove "The Gamers Benchmark" slogan and everything should be pretty much clearer. The smart folks really know what 3DMarkXX means.
Joe DeFuria
20-Aug-2003, 17:23
Joe, remove "The Gamers Benchmark" slogan and everything should be pretty much clearer. The smart folks really know what 3DMarkXX means.
Rev, I agree that everything is pretty clear from the whitepaper. I wholly supported it and had high praise for it actually. I spent many hours on this very message baord explaining it to those who didn't "get it."
That doesn't change the fact that what's clear to me in the white-paper, doesn't jive IMO with the allowing for "optimizations" and such.
In any case, I would really like to hear some more subjective and direct answers to the questions I posed, because again, given recent events I am not convinced that what was said in the white-paper, still holds true today.
]Out of curiosity, what about all games with built in benchmarks?
Worm,
thanks for taking the time to respond. But I think I made my point clear this was in regards to 3dmark. For games. Then I agree with DT. So long as NO IQ change is seen then its acceptable. However targeting only the big benchmark games and only optimizing the standard time demos is very wrong. I hope you can agree to that.
However a benchmark like 3dmark is giving the video card a set order, a set task and a SET amount of work. If an IHV optimizes then you change the amount of work and that's kills any base line you can make for a common comparisons. To make a valid comparison you need a solid base line. If some company says games don't do it that way. Fine, then USE THAT GAME to test as well. But don't allow optimizations just because game X does. In fact we have seen from Q3 that games based of its engine and video cards FPS scores don't correlate at all (ie look at any review with Q3 and one of its other games based off it and you will see wildly different results). EVERY game out there is going to be different. If you make the change now and allow any optimizations then you have defeated in making a true benchmark.
fallguy
20-Aug-2003, 18:53
If FM would only let WHQL drivers in their database, it would sure hurt nVidia. They said something about that months ago, I guess they were convinced not to do it.
PatrickL
20-Aug-2003, 19:01
Whql does nothing to prevent cheats in drivers.
fallguy
20-Aug-2003, 20:34
Most of the time it seems, the cheats are mainly in the leaked drivers. The top scores are all with leaked drivers.
Myrmecophagavir
20-Aug-2003, 21:09
]AFAIK There have been issues with certain chipsets reporting as "the same product" to the Database. In other words, a Radeon 9800 and a Radeon 9800 Pro might both be reporting and grouping themselves as "9800". Same with nVidia cards.
This might be a re-vamp to try and actually separate differnt SKUs of the same chip? So you can distinguish a GeForce 5900 Ultra from a 5900, or a ATI 9500 from a 9500 Pro, etc.
That was already done in the "old" HoF. We already got the (for example) Radeon 9800 and Radeon 9800 Pro, 9500 series and 9700 series all separated etc.
But it obviously didn't work for all cards, did it. The 9500 was there in the DX9 list, but if the 9500 Pro was listed separately it was out of the top 10 :!: :? Same for the 9000. And the 9600, 9100, 8500 and 7500 were just "Series". If you're going to do it, you have to be thorough. If ATI now has SE variants going around too, will those be separately indexed? And what are you going to do about bus-width-crippled versions of cards, if they're not labelled nicely?
And please tell me you'll actually look at the results before publishing them this time and do a sanity check, to prevent things like the GF2 Ultra's former position... :roll:
Chris123234
20-Aug-2003, 22:00
And please tell me you'll actually look at the results before publishing them this time and do a sanity check, to prevent things like the GF2 Ultra's former position...
What happend with the GF2 Ultra's former position?
Reverend
21-Aug-2003, 02:39
Joe, remove "The Gamers Benchmark" slogan and everything should be pretty much clearer. The smart folks really know what 3DMarkXX means.
Rev, I agree that everything is pretty clear from the whitepaper. I wholly supported it and had high praise for it actually. I spent many hours on this very message baord explaining it to those who didn't "get it."
That doesn't change the fact that what's clear to me in the white-paper, doesn't jive IMO with the allowing for "optimizations" and such.
In any case, I would really like to hear some more subjective and direct answers to the questions I posed, because again, given recent events I am not convinced that what was said in the white-paper, still holds true today.
Like Patric had said (perhaps a couple of times) in our forums, FM is working on determining the considerations for both "absolutely no optimizations" as well as "governable optimizations" (both quoted phrases are my own words). I doubt FM personnels would explain things further regarding this when the subject matter is a WIP at FM currently.
PS. My own official (as a beta member) contribution/submission-of-opinion to the subject matter more or less agrees with what FM already has in mind, at least that's what Tero told me. I gave my answers the day after I received the doc from FM... I already know what I want in the next 3DMark.
I already know what I want in the next 3DMark
I look forward to seeing the next 3DMark, I have had alot of fun with all of the versions....
Nick[FM]
21-Aug-2003, 12:28
But it obviously didn't work for all cards, did it. The 9500 was there in the DX9 list, but if the 9500 Pro was listed separately it was out of the top 10 :!: :? Same for the 9000. And the 9600, 9100, 8500 and 7500 were just "Series". If you're going to do it, you have to be thorough. If ATI now has SE variants going around too, will those be separately indexed? And what are you going to do about bus-width-crippled versions of cards, if they're not labelled nicely?
We separated all cards we could. If you check the inf file of any displ driver, you should be able to see what is possible and what is not. AFAIK the Radeon9600 & Pro are listed only as "RADEON 9600 SERIES". Also, the 9500 Pro is combined with the 9700 as "RADEON 9500 PRO / 9700". There's no way to detect which is which if the driver doesn't tell us. Take a peek at the driver inf files and you will see what I mean. As soon as they separate them in the drivers, we can separate them in our online services. We are constantly updating as new drivers are released. Hope this shared some light on the subject?
And please tell me you'll actually look at the results before publishing them this time and do a sanity check, to prevent things like the GF2 Ultra's former position... :roll:
We always take a look at the lists before they are posted. Keep in mind that the lists are based on user submitted results. We don't "make" the lists ourselves. They are generated based on the data we have. But now the lists are down due to maintenance. Let's hope that all your worries will be gone when the new and improved lists will be up! :wink:
digitalwanderer
21-Aug-2003, 16:10
]
We always take a look at the lists before they are posted. Keep in mind that the lists are based on user submitted results. We don't "make" the lists ourselves. They are generated based on the data we have.
Thanks for taking the time to post up here, but are you saying that the list is manually made or is it just a generated list...for some reason I'm confused. (Sorry, long night. :roll: )
Nick[FM]
22-Aug-2003, 09:23
Thanks for taking the time to post up here, but are you saying that the list is manually made or is it just a generated list...for some reason I'm confused. (Sorry, long night. :roll: )
The list is generated. It is not made manually. We only take a look at (sort of preview) it before it goes online.
just me
22-Aug-2003, 23:25
Glad to hear FM is 'fixing' the HoF, just wish they'd have said the same things in my post(s) @ FM about these issues. Good thing I come here to find out what is happening on the site I frequent the most > FM. :roll:
Rev, interesting comments. Maybe an intellectual debate betrween you & Neeyik on "The Gamers Benchmark" claim. :wink: I would find it very interesting & informative.
So I guess as it stands today (Aug 22, '03) 3DM03 usage, the HoF, etc. is 'undecided'? I read Patrics' comments in his Forum about moving forward, but w/the future soo unclear, how can we? Maybe once FM makes a decision (& sticks to it this time) we'll have a guide to lead us forward. :wink: I can only hope. 8)
.02,
Nick[FM]
25-Aug-2003, 09:39
Glad to hear FM is 'fixing' the HoF, just wish they'd have said the same things in my post(s) @ FM about these issues. Good thing I come here to find out what is happening on the site I frequent the most > FM. :roll:
I simply answer questions here (the ones I can answer). Same thing at our own boards. The problem is that our boards are pretty huge, and therefore some questions might slip thru unanswered. If you ever need to know something important or want an answer quickly, you can always send me a PM over at our boards (or here).
digitalwanderer
25-Aug-2003, 15:05
]Thanks for taking the time to post up here, but are you saying that the list is manually made or is it just a generated list...for some reason I'm confused. (Sorry, long night. :roll: )
The list is generated. It is not made manually. We only take a look at (sort of preview) it before it goes online.
Ah, you do have a sanity check then. Thank you. :)
...Like Patric had said (perhaps a couple of times) in our forums, FM is working on determining the considerations for both "absolutely no optimizations" as well as "governable optimizations" (both quoted phrases are my own words). I doubt FM personnels would explain things further regarding this when the subject matter is a WIP at FM currently.
The problem is that FM has already made this decision, and explained it, in their original Audit Report .pdf in which they stated their case very persuasively.
If we apply the prinicple of Occam's Razor here (pardon my spelling if it's off), FM got it absolutely right the first time. There shouldn't be *any* benchmark-specific optimization going on, because 3dMk03 is a benchmark, not a 3d game--it doesn't even pretend to be a game. Hence, all of nVidia's optimization comments relating to 3d games simply do not apply.
The simplest, easiest, and best--completely fair--approach for FM is to tell the IHVs that no 3dmk03-specific optimizations (and certainly cheats) will be tolerated. Period. This solves the problem of undue complexity and places everybody on the same level playing field. There simply is no reason whatever to rethink their original audit report--at least, any reason having to do with the creation of an impartial benchmark. IMO, the problem here is one inherent in FM's business model, not in logic.
digitalwanderer
25-Aug-2003, 16:39
There simply is no reason whatever to rethink their original audit report
Well, 'cept the almighty dollar... :(
demalion
25-Aug-2003, 21:47
There is indeed a case for 3dmark specific optimizations being part of the benchmark. It just makes it, in my evaluation, a far less useful benchmark. It becomes "just another game", except with more tools for data analysis (for example, sort of like UT2k3 with its rich, but different, benchmarking featureset and API feature focus).
My argument for this (http://www.beyond3d.com/forum/viewtopic.php?p=121642&#121642). I also feel Dave H's argument, that I refer to there and which can be found by scrolling up, reflects my viewpoint closely, in a way less dependent on context.
There is indeed a case for 3dmark specific optimizations being part of the benchmark. It just makes it, in my evaluation, a far less useful benchmark. It becomes "just another game", except with more tools for data analysis (for example, sort of like UT2k3 with its rich, but different, benchmarking featureset and API feature focus).
My argument for this (http://www.beyond3d.com/forum/viewtopic.php?p=121642��). I also feel Dave H's argument, that I refer to there and which can be found by scrolling up, reflects my viewpoint closely, in a way less dependent on context.
I would say that it might make 3dMk "just another game which has been heavily optimized for," instead of "just another game," because the bulk of 3d games don't receive that kind of optimization attention--probably 95% of them. Here's a good example of the way most games will run, I think:
http://www.beyond3d.com/misc/traod_dx9perf/
Considering the shader-PS2.0-DX9 component, I think it tends to underscore why FM should stick to its original plan and outlaw all optimization for the benchmark--which would make it more likely, rather than less, to indicate actual 3d-game performance.
demalion
26-Aug-2003, 03:47
Heh, please read the provided link, Walt.
I'm pointing out that specific optimizations do not automatically make it useless as a benchmark. Notice how suitable my UT 2k3 parallel is to your concerns, and also how related it is to the related issues of what reviewers expose to consumers in their reviews and the burden of evaluating what the actual optimizations accomplish.
3D Mark is not a 3d game--it doesn't even pretend to be a game.
In that case, why call the four tests that make out the final score "game tests" ?
These tests also includes physics (at least AFAIK) just as a real game would do. This doesn't make 3D Mark a game but i wouldn't say that it didn't in any way pretend not to be.
StealthHawk
26-Aug-2003, 08:25
I'm pointing out that specific optimizations do not automatically make it useless as a benchmark.
It depends on what kind of optimization it is. One of the optimizations NVIDIA is doing is replacing shader code with less stressful shader code. When only one IHV is doing less work, this is clearly unacceptable for a benchmark- it makes the results totally useless.
OTOH, optimizing code for register usage and making the output stay the same may not necessarily be bad, although IMO any application specific optimizations in a synthetic benchmark are iffy at best. For a benchmark like 3dmark which relies on the Orb database for users to compare their scores, optimizations weaken the benchmark's usefulness.
In that case, why call the four tests that make out the final score "game tests" ?
These tests also includes physics (at least AFAIK) just as a real game would do. This doesn't make 3D Mark a game but i wouldn't say that it didn't in any way pretend not to be.
I don't understand your point--we both agree that 3dMK03 isn't a 3d game--it's a benchmark. It can't be played--only run. Regardless of what FM calls the specific tests--they can't be played--so where's the gaming element? Conversely, what 3d game do you know of which functions like 3dMk03?...;) (IMO, it would be a very boring game, indeed.)
I wasn't aware that there was any confusion as to the nature of 3dMK03...if it *is* a 3d game--I'd love to learn how to play it...;) As it is I only know how to run it as a benchmark.
I don't understand your point--we both agree that 3dMK03 isn't a 3d game--it's a benchmark. It can't be played--only run. Regardless of what FM calls the specific tests--they can't be played--so where's the gaming element? Conversely, what 3d game do you know of which functions like 3dMk03?...;) (IMO, it would be a very boring game, indeed.)
I wasn't aware that there was any confusion as to the nature of 3dMK03...if it *is* a 3d game--I'd love to learn how to play it...;) As it is I only know how to run it as a benchmark.
Yes, it's rather obvious that 3D Mark is not a game and doesn't pretend to be a game in that sense. But as i see it, it definitely tries to resemble a game afa benchmarking goes and that's why i thought the "it doesn't even try to resemble a game" was not entirely correct.
demalion
26-Aug-2003, 22:04
I'm pointing out that specific optimizations do not automatically make it useless as a benchmark.
It depends on what kind of optimization it is. ...
Err...exactly. Heh, I guess you're not as familiar with my viewpoints on such issues as I am with yours, which is something that I think will be addressed if you follow my initial link (I don't like re-typing my commentary again, and it should be just as easy to read there as here. :) ).
Myrmecophagavir
27-Aug-2003, 01:09
]The list is generated. It is not made manually. We only take a look at (sort of preview) it before it goes online.
Ah, you do have a sanity check then. Thank you. :)
Well hurrah and all, but, you know, what happened the first time round then?
Seen in FM latest news (top article on their front page):
]http://www.futuremark.com/news/?newsarticle=200308/2003082904#200308/2003082904:
So, we look only at the FX5600Ultra and 9600Pro - the view is marvelous! From that it's absolutely clear why the slightly overclocked revision to FX5600Ultra was released. The 10% performance boost allowed to win the crown back to NVIDIA at the overall scores.
What's the point of retiring the flawed HoF if it's to publish such biased snippets on the other hand? worm conveniently omitted the conclusion of the article: "Even with the slower (and thus cheaper) memory, the RV350-based video card (600 MHz versus 800 MHz) demonstrates some superiority over the NV31 new core."
'nuf said...
Doomtrooper
30-Aug-2003, 12:25
http://smilies.crowd9.com/otn/happy/mblah05.gif
Seen in FM latest news (top article on their front page):
]http://www.futuremark.com/news/?newsarticle=200308/2003082904#200308/2003082904:
So, we look only at the FX5600Ultra and 9600Pro - the view is marvelous! From that it's absolutely clear why the slightly overclocked revision to FX5600Ultra was released. The 10% performance boost allowed to win the crown back to NVIDIA at the overall scores.
What's the point of retiring the flawed HoF if it's to publish such biased snippets on the other hand? worm conveniently omitted the conclusion of the article: "Even with the slower (and thus cheaper) memory, the RV350-based video card (600 MHz versus 800 MHz) demonstrates some superiority over the NV31 new core."
'nuf said...
Well, when you consider what they ran on Thursday:
http://www.futuremark.com/news/?newsarticle=200308/2003082805#200308/2003082805
...it's not so bad...;)
John Reynolds
30-Aug-2003, 16:31
I wonder if ATI management is considering dropping out of FM's beta program now?
I wonder if ATI management is considering dropping out of FM's beta program now?
Frankly, I don't understand the IHVs' attraction to it at all. IMO, I think both of them far over-estimate its importance to them in selling 3d cards. Perhaps they use it as a vehicle to guess each other's future directions and implementations...?
John Reynolds
30-Aug-2003, 16:53
The September issue of Maximum PC has their Dream Machine of 2003 article and what graphics card do you think they picked and based on what criteria? And do you think it really scored over 7,000 in 3DMark03 (page 29)? How many people will read that article and think: "That's the card to buy this fall!!!"
martrox
30-Aug-2003, 16:58
While thinking about what motivates IHV's in concert to FM, think about this: Just WTF is (and has) been going on with FM in concert to IHV's? These guys have been all over the place. First, they were nVidia's darling, back in the original GeForce days through and including the GF4's. Then, in a bid to finally appear objective, FM decides to grow some cajonies and actually start to support DX, rather than just one IHV :wink: . Then, just when FM looks the part of a real hero, they cut off the cajonies and becomes nVidia's whore again. It's pretty incredible to believe these guys will ever be taken seriously again. Maybe FM should go into politics...... they certianly have the correct moral fiber for it.
Bouncing Zabaglione Bros.
30-Aug-2003, 17:16
I've given up on 3Dmark - not because of the technical aspects of the benchmark, but because of the attitude of Futuremark and their inability to remain objective, or to stick with the independent ideals they profess to have. IMNSHO, FM are an unreliable, untrustworthy company, ever since they decided to throw their lot in with the unreliable, untrustworthy Nvidia.
I've had Futuremark products on my PC since day one, and for years I've had their software on my hard drive. Not any more.
The September issue of Maximum PC has their Dream Machine of 2003 article and what graphics card do you think they picked and based on what criteria? And do you think it really scored over 7,000 in 3DMark03 (page 29)? How many people will read that article and think: "That's the card to buy this fall!!!"
But John, this assumes that paper trade mags like MPC are the only source of information for the bulk of its readership. Obviously, the rise of the Internet--indeed, its ubiquity--over the last few years has put a serious crimp in the authority and influence of the paper trade mags in the market. Just because a paper mag recommends this or that no longer equates to an automatic assumption that such recommendations will in fact be heeded by even a majority of its own subscription base. Why? Because that subscription base is also looking at the Internet even while it reads the magazine. So the question is one of whether any advice provided in a paper mag will be heeded if it is contrary to what its subscription base is reading on the Internet.
The biggest problem for paper trade mags going forward is the lead time. Whereas product news on the Internet is observable almost as it occurs, paper mags often highlight info that is as much as 60 days old when it arrives in your mailbox. This was A-OK and the status quo before the ubiquity of Internet access for most people--especially people interested in the purchase of computer peripherals. But IMO these days depending on paper trade mags for product info is very nearly passe' and almost always redundant. Paper mags have been able to offset that advantage to some degree by offering CDs full of patches and demos which are still attractive to people who do not yet have broadband. I dropped my last paper subscription a couple of years ago when I went to broadband and, indeed, the only reason I continued to subscribe to paper mags up to then was because of the CDs which were valuable to me when I was doing 56K telephony. Otherwise, I'd have dropped paper subscriptions 3-4 years ago. This isn't to say that paper mags have no influence--I'm sure they do. But I'll wager that influence has been diminished a great deal from what it was just four years ago.
Another problem with the paper mags is that besides being late with the information, the quality of the information rarely is comparable with that available on the Internet from a variety of sources. Then there's the fact that with falling subscriptions the ratio of advertising to content has been skewed such that sometimes there are more ads than articles--and also that the number of pages in paper publications seems to continuously decline. As such, I'd say there's a good chance that what people would say instead is, "I know they recommend this card, and I've read several supporting articles on the Internet about that card already, so I'll buy it," or "They recommend this card but I've read too many negative reviews on these products on the Internet, so I'll pass."
More realistically because of the time lag relative to the Internet, I can see people going ahead and making a choice and buying their 3d cards before the magazine making its recommendations even arrives in their mailboxes...;)
Bouncing Zabaglione Bros.
30-Aug-2003, 18:37
As Waltc says, the sort of computer-geek who's going to drop $400 on a graphics card is already hooked up to the net and has already done the research online. Nowadays, anyone who makes their buying decisons based on what the paper magazines say isn't the sort of person buying $400 graphics cards.
The people I know who do still read the mags would not be in the market for that kind of product, and if they were, they'd be asking *me* for recommendations on what to buy before trusting their money to a magazine review.
digitalwanderer
30-Aug-2003, 19:11
The people I know who do still read the mags would not be in the market for that kind of product, and if they were, they'd be asking *me* for recommendations on what to buy before trusting their money to a magazine review.
Agreed. I can't count how many times someone has asked me for a recomendation because the PR on cards can be over-whelming after a while.
I think the print media is dying. It ain't dead yet, but the end is in site.
It ain't dead yet, but the end is in site.
as in websight? ;)
John Reynolds
30-Aug-2003, 21:23
But John, this assumes that paper trade mags like MPC are the only source of information for the bulk of its readership.
Walt, there've been too many studies done that show how easily influence people are by things they read in print.
But John, this assumes that paper trade mags like MPC are the only source of information for the bulk of its readership.
Walt, there've been too many studies done that show how easily influence people are by things they read in print.
I agree, but I was only saying they can read the Internet print, as well...;)
Fred da Roza
30-Aug-2003, 22:54
I wonder if ATI management is considering dropping out of FM's beta program now?
As I've said in the past ATI leaving won't help their bottom line or diminish the misperceptions people have because of FM. Although cheating may not be the most ethical thing to do, ATI should. If FM won't enforce a level playing field, ATI should level it for them. Until FM has a financial reason to act, they won't. And at this point should anyone care about the well being of FM.
I wonder if ATI management is considering dropping out of FM's beta program now?If FM won't enforce a level playing field, ATI should level it for them. Until FM has a financial reason to act, they won't. And at this point should anyone care about the well being of FM.
I'm also thinking that cheating will be much complicated for ATI that for nVidia. Clipping-Planes apart, nVidia could easyly gain some performance rewriting shaders for different registry use, or forcing partial precision. Those tricks won't work for ATI.
But surely there is something that can be done, sacrificing some IQ.
Maybe ATI could do some work on a next version of Catalyst and then come out loudly saying that they ARE optimizing specifically for 3DMark, for providing a leveled playing field.
Or may be they could try the plug-in way, as I humbly hipotized here (http://www.beyond3d.com/forum/viewtopic.php?t=7647) (is that such a bad idea?).
Edit: Never mind... probably that is really a bad idea! :)
Bye!
I m a little surprised that S3/VIA is not a Beta member of FM (http://www.futuremark.com/betaprogram/)anymore,I remember S3/VIA is on the list just one month back.
Does S3/VIA quit FM Beta program?
Hope Worm or Patric Can confirm it.
CorwinB
04-Sep-2003, 14:18
I'm wondering if getting S3/VIA back on the (rocking) beta member boat will be a new "top priority" for FutureMark... :shock:
I wonder if ATI management is considering dropping out of FM's beta program now?If FM won't enforce a level playing field, ATI should level it for them. Until FM has a financial reason to act, they won't. And at this point should anyone care about the well being of FM.
I'm also thinking that cheating will be much complicated for ATI that for nVidia. Clipping-Planes apart, nVidia could easyly gain some performance rewriting shaders for different registry use, or forcing partial precision. Those tricks won't work for ATI.
But surely there is something that can be done, sacrificing some IQ.
Maybe ATI could do some work on a next version of Catalyst and then come out loudly saying that they ARE optimizing specifically for 3DMark, for providing a leveled playing field.
Or may be they could try the plug-in way, as I humbly hipotized here (http://www.beyond3d.com/forum/viewtopic.php?t=7647) (is that such a bad idea?).
Edit: Never mind... probably that is really a bad idea! :)
Bye!
They should just remove 95% of the colour, say it's a driver bug, and anyway they like benchmark noir. That ought to bump up the scores.
They should just remove 95% of the colour, say it's a driver bug, and anyway they like benchmark noir. That ought to bump up the scores.
Yeah! Black is better for benchmarking. It leave out unmeaningful IQ questions, freeing the raw power!
:D :D :D :D
Bye!
K.I.L.E.R
05-Sep-2003, 07:51
Wireframe rendering is better. :D
Who needs textures anyway? ;) :lol:
parhelia
05-Sep-2003, 09:07
I m a little surprised that S3/VIA is not a Beta member of FM (http://www.futuremark.com/betaprogram/)anymore,I remember S3/VIA is on the list just one month back.
I guess S3 wants to cheat in their drivers....
Anyone knows why both XGI and Trident are listed in the beta program? I thought XGI bought Trident.
vBulletin® v3.8.6, Copyright ©2000-2013, Jelsoft Enterprises Ltd.