GF4 has inflated 3dmarks scores so says the INQ.....

[noko]

Benches are great for tweaking ones performance but not the best when compairing different hardware. Just a limitation of benchmarking which is only worsen if a manufacturer writes specific code to nudge the results in their favor.

I remember a number of reviewers never would use 3dMark for testing. Looks like MadOnion convince them otherwise in the last year or so, now I think reviewers who are compairing different hardware should stay clear of 3DMark due the apparent ease of optimizing around it vice optimizing in general. I remember a number of people and myself noted that our 3dMark score went up but yet our game performance stayed the same or went down. I rather have optimization in the games and programs I play instead of some pseudo game tests.

 

[Matt Burris]

I just post up the default score in 3DMark 2001SE for my reviews. I just don't see the point of using a synthetic benchmark to show readers what it can do for you in games. It gives a good gauge of what you can expect in future games, but as a comparitive tool, I don't see its effectiveness. When I see a 3DMark score, nothing really goes through my mind. When I see a score in Quake 3, I can relate to it, or Serious Sam, because I play those games.

 

[Doomtrooper]

Matt I do remember you in the old forums ranting about how you couldn't break 8000 ( I think)on your Geforce 3, and were asking for input...
Downplaying the importance of this benchmark is a joke, there is guys on the ORB that spend thousands of dollars on hardware for every little point, it is packaged with some Nvidia cards as part of the software bundle..every forum is filled full of Signatures linking 3Dmark scores..cooling companies get rich by people buying water cooling solutions for every point ??
People base their buying decision on 3Dmark...I see nobody standing up and taking any blame for this either, Madonion or Nvidia..who is at fault..or both

 

[Neeyik]

Why should anyone be taking some blame here, when nobody is completely sure as to why it is happening? MadOnion have stated that they believe it to be a driver problem but they're also investigating it.

People really need to stand back at this whole thing and look at the sequence of events. When the bug was first noticed, the person who discovered it never made any mention of the word cheating in his forum post. More importantly, not that many other people replied with similar threads and as newer cards and drivers appeared over the months, the knowledge of this bug seemed to disappear. Look at it from MadOnion's point of view: a user points out this error using beta drivers. After a while, nobody says anything about it. Other cards don't have the problem. MO don't have racks of equipment testing every single leaked beta driver that floats onto the net and I should well imagine in this case, they thought that it was almost certainly down to the drivers.

Time has passed on but the problem has appeared again. This time, the threads are awash with accusations of NVIDIA cheating or MadOnion being completely biased towards NVIDIA. Why? Because they are the "in things" at the moment. Yes, other cards seem to have the problem but it's not consistent and it's certainly not even consistent as to which game tests take a performance hit.

 

[Entropy]

Matt I do remember you in the old forums ranting about how you couldn't break 8000 ( I think)on your Geforce 3, and were asking for input...
Downplaying the importance of this benchmark is a joke, there is guys on the ORB that spend thousands of dollars on hardware for every little point, it is packaged with some Nvidia cards as part of the software bundle..every forum is filled full of Signatures linking 3Dmark scores..cooling companies get rich by people buying water cooling solutions for every point ??
People base their buying decision on 3Dmark...I see nobody standing up and taking any blame for this either, Madonion or Nvidia..who is at fault..or both

IMO both are at fault. nVidia for doing this in an easily detectable way, and MadOnion for knowing about it without having the strength to tell nVidia to stop doing it so people can see.

In the case of 3DMark, nVidia can't really cheat because MadOnion have NO rules against application specific driver behaviour, nor any criteria for correct output. Add to this that MadOnion is not a non-profit organisation, and they have close ties/dependencies/mutual interests with those whose products they test. Now, anyone who knows these things can think it through for himself and come to a conclusion about how important a 3DMark score is, even besides the fact that it is a benchmark that doesn't directly map to the performance of any other application.

Problem is, as a rule consumers don't seem to think these things through, and nor do they get much help from special interest sites and magazines. (People actually like single figures of merit, it helps with making the world a less complex place. "How fast is my computer = 3DMark score".)

In this situation it seems likely that nVidia will see to that the behaviour doesn't show up, and that the issue will either be denied or ignored until this storm in various forums have blown over. Which will take no more than another day or so. Remove the irritant, smooth over a little if need be, and the world will go on much as it has.

Entropy

 

[Sharkfood]

This subject does bring about some interesting topics.

Let's say, for the sake of argument, that all is fair game with 3DMark. He/she that can obtain the highest scores may do so however they see fit, even if such optimizations or tweaks will NOT have any applicability outside of 3DMark.

What does this do to MadOnion? Is their benchmark assumed to have a bearing or resemblence to the performance of 3D hardware outside this application? If there is even the slightest *intent* that 3DMarks are to be used to measure "3D performance", and not simply "3DMark Performance" then this kind of finding would obviously invalidate such intent.

If 3DMark is supposed to obtain value outside it's own specific realm of conditions, then such a finding would be derrogatory to it's purpose.

So what we have here really is not only a noted behavior of uncertainty of motive/cause, but also a benchmark that leaves uncertainty to it's purpose to whether or not such a thing is "okay" by intent, or just something some people might consider "immoral."

Obviously, if a driver team discovers a few lines of code that can be added that will improve 3DMarks by say 5-8%, but will have absolutely zero effect outside of 3DMark would pose a situation. The only governing principle would be one of ethics to start, but this could easily be countered by arguments for the purpose of the benchmark (i.e. not a stated/implied intention of being a useful measurement of anything else EXCEPT 3DMark2001). Therein lies the conflict.

Also obviously, gamers and consumers are not persuaded by such arguments as too many "hardware" reviewer opinions put pretty big weight on 3DMark scores as an indication of 3D performance. Most printed and web form reviews calculate their total "product" scores based almost majority weighted on 3DMarks. But who is "wrong" in this case? Are the reviewers incorrectly assuming the purpose of the tool? Or is the tool simply marketed or implied to have usage outside it's delivered goal?

Perhaps where the real solution lies is getting the intent/mission of the benchmark clearly stated by MadOnion (and regulated if this intent is to hold water or credibility), followed by "professionals" using this tool to ensure proper understanding of it's purpose.

Just my $0.02,
-Shark

 

[Stormlord]

3Dmark

Well, I'll have to say one thing. The people at Mad Onion sure make a nice demo As to the usefulness as a benchmark: beats me. That's the way they 're marketing it, but what if you have to depend on a third party's drivers to provide you with the results & they tweak the results returned by the driver based on the app that's using it? If they really want to have it produce accurate results they would probably have to low level program to the hardware directly(and so circumvent any software made by the manufacturer). Which they will probably never do, because they might not have all the info they need & it just takes up way to much work to code their own limited driversets for each chip & hardware out there. Plus it 's a kind of a job not too many programmers are capable of & willing to spend time on these days (low level assembler programming).

There is no such thing as a perfect benchmark & the only way they would be able to do it is to entirely forget about directx, opengl & other third party stuff if they want to produce more or less reliable results.

3Dmark results do have a serious impact on sales of various hardware, we would be naïve to ignore that. However, people should be more sceptical about benchmarks & first find out what they REALLY need before buying tons of hardware based on a benchmark score. It's all about bragging-rights I guess...

 

[Ailuros]

It's all about bragging-rights I guess...

I don't disagree with the points made by other posters so far in general, yet this sentence deserves to be reposted.

 

[Entropy]

Let's say, for the sake of argument, that all is fair game with 3DMark.

You don't have to be so cautious.

Comparing to SPEC, the situation is exactly as if vendors could use any subtest specific tricks within the compilers, the output wasn't checked for correctness, and the whole show was run by BAPco .

3DMark is wide open for manipulation on all sorts of levels.

Entropy

 

[noko]

I've found 3DMark is too limited to really test between two cards capability. Reason being is that each card for one could have a drastically different Texture Level of Detail which affects the final results. I remember the GF2 with its notorious suck ass texture compression and the Radeon was putting out significantly better images, yet 3DMark was oblivious. 3DMark fails to control a number of aspects of a video card to give an accurate comparitive result. It maybe pretty and yes I very much look forward in the next kick ass version but for benchmarking it is best to use the actual program or game you are going to use to compare two different cards.

Oh yea, to do something to inflate numbers for a better showing but yet has no beneficial result to anything else except for one particular benchmark test, well in my book that is CHEATING.

 

[Neeyik]

Please do correct me if I'm miles off the mark here but doesn't the general order of things go:

application > API (and drivers) > HAL > hardware

How is 3DMark supposed to get around this? Surely it's the fact that the drivers can be tweaked to hell and back that raises doubts of the results from ANY 3D benchmark, and not just MadOnion's.

Oh and criticising 3DMark on the basis that it ignores the quality of the 3D image for the scores is a bit of weak one. Do please explain how Q3A timedemos do this, as they get used equally as well as 3DMark does, in magazine reviews of video cards. Come to think of it, look at the number of web reviews that use Q3A...

 

[Doomtrooper]

Neeyik,

Quake3A is a game, you can play it online, you can go to a store and buy if off the shelf, it was marketed as a game and it is a GAME.
3Dmark is a benchmark, you can't play in online and finding it on a store shelf is very difficult, it is marketed as a BENCHMARK utility.

If a certain brand of video card can gain approx 200-800 points by disabling a splash screen and no others do then I'd say there is a problem

 

[Entropy]

Please do correct me if I'm miles off the mark here but doesn't the general order of things go:

application > API (and drivers) > HAL > hardware

How is 3DMark supposed to get around this? Surely it's the fact that the drivers can be tweaked to hell and back that raises doubts of the results from ANY 3D benchmark, and not just MadOnion's.

You are not off the mark. As far as I can see you are completely correct. And that's the problem.

Look, neither nVidia or MadOnion can be, or is, (at least by me) accused of any wrongdoing.
MadOnion never claimed that they had any rules for drivers, nor that they applied any criterion for correctness of output. And they obviously don't hide that they are in the business of making money.
nVidia are fully within their rights to optimise and adjust their drivers in any way for any application they damn well please. And they would have to be exceedingly unprofessional not to do it for 3DMark, since it has such great impact on percieved value of their products, and their brand.

Entropy

PS. Q3 is a game, that can be used for benchmarking. 3DMark is a product meant to be a benchmark, and nothing else. There is a huge difference between the positions of id and MadOnion.

 

[noko]

Oh and criticising 3DMark on the basis that it ignores the quality of the 3D image for the scores is a bit of weak one. Do please explain how Q3A timedemos do this, as they get used equally as well as 3DMark does, in magazine reviews of video cards. Come to think of it, look at the number of web reviews that use Q3A...

Exactly, since in its present form it can't accurately determine what is being rendered then calling it a 3D benchmark when the output could be pure crap (GF2) or then optimize around makes it no longer a benchmark for compairing cards. Its like compairing two cars in how fast they can go, in car one you have it going down hill while car 2 it is going up hill at a 45 degree angle. Yes you can get the acceration and speed of both cars but compairing them against each other in this case is just B.S.. That is what 3DMark is doing there buddy, the results are not always compariable. Sounds like if Madonion wants to call it a benchmark then they should make it a benchmark.

QuakeIII is a game, sold as a game not labelled as a benchmark big difference. Now if people use it as a benchmark then you come across the same problems wouldn't you.

To assume that if the GF4 gets a higher 3DMark2001 over a Radeon means that it will be faster then application X over here is totally false. It turns out in certain conditions that the Radeon8500 beats even a GF4 Ti4600 . So much for 3DMark, it has bloated into something that it isn't. It isn't a test of overall 3d performance. It is a test of points, points only.

 

[Ichneumon]

In my mind I'm not really interested in the "cheating" aspect of it as I am in finding out whether or not Nvidia really is doing game-test specific driver optimizations.

Perhaps in the next couple days I'll sit down and see about perhaps changing the names and/or images for the splash screens. I don't have a GF to test it though... and i'm not sure i have the l33t skillz anymore to accomplish it. Will just have to see.

 

[Doomtrooper]

Please do correct me if I'm miles off the mark here but doesn't the general order of things go:

application > API (and drivers) > HAL > hardware

How is 3DMark supposed to get around this? Surely it's the fact that the drivers can be tweaked to hell and back that raises doubts of the results from ANY 3D benchmark, and not just MadOnion's.

You are not off the mark. As far as I can see you are completely correct. And that's the problem.

Look, neither nVidia or MadOnion can be, or is, (at least by me) accused of any wrongdoing.
MadOnion never claimed that they had any rules for drivers, nor that they applied any criterion for correctness of output. And they obviously don't hide that they are in the business of making money.
nVidia are fully within their rights to optimise and adjust their drivers in any way for any application they damn well please. And they would have to be exceedingly unprofessional not to do it for 3DMark, since it has such great impact on percieved value of their products, and their brand.

Entropy

PS. Q3 is a game, that can be used for benchmarking. 3DMark is a product meant to be a benchmark, and nothing else. There is a huge difference between the positions of id and MadOnion.

You had better tell Daid Kirk that then

"We at Nvidia don't make it a practice to optimize our pipeline for specific benchmarks - we want to provide high quality and high performance on a wide variety of useful and entertaining applications, rather than just getting a good score."

David Kirk, Nvidia

I'm also not interested in the cheating aspect I want to know what is gonna be done with the Orb database.

 

[Crusher]

I've found 3DMark is too limited to really test between two cards capability. Reason being is that each card for one could have a drastically different Texture Level of Detail which affects the final results. I remember the GF2 with its notorious suck ass texture compression and the Radeon was putting out significantly better images, yet 3DMark was oblivious.

Here we go again...

Have you even seen a GeForce 2 run 3DMark2001SE using texture compression? It looks just fine. The "notorious suck ass texture compression" is not suck ass anymore, if it ever was. And do you have any idea what the difference is in 3DMark2001SE score between mipmap detail set to "Best Image Quality" and "Best Performance"? A whopping 25 points. And lest we forget, mipmap LOD adjusting is not specific to NVIDIA. In fact, as I recall it wasn't their idea to introduce such a thing into the world of 3D graphics cards, and they only did so in response to 3DFX's use of the technique when they were attempting to find any way possible to make their products appear to have better quality than their competitors. So if you want to blame anyone, blame 3DFX.

 

[Neeyik]

I distinctly remember a thread, not in the too distant past, that had countless people arguing over whether ATI cards were rendering the game tests "properly" based on the Image Quality tests in 3DMark. Having NVIDIA cards getting a score advantage due to users being able to adjust the LOD bias (and therefore having awful looking images) isn't MadOnion's problem - that's an issue for the video card manufacturers. If ATI wanted a level playing field on that issue, then all they need to do is allow LOD bias adjustments by the user in the driver panels.

So for nobody has successfully argued that Q3A is never used as a benchmark to review video cards with and therefore assess their viability as a purchase. Magazines and websites do it alike - it is game, yes you are all correct on that, and no 3DMark isn't but they both get used to benchmark, test and compare video cards and therefore influence the views of the buying public.

A GeForce4 4600 only gets a score a small amount higher than a Radeon 8500 - or at the very least, it does in my machine - and in all the D3D games I've compared to the with:

Max Payne
Morrowind
Dungeon Seige
IL-2 Sturmovik

...the GF4 is faster than the Radeon 8500. Not excessively so but the point is, the card gets a higher score in 3DMark (about 800 points more) and the above games run faster too. It's faster in OpenGL games too but there isn't a universal OGL benchmark that both cards have a level playing field in.

If a certain brand of video card can gain approx 200-800 points by disabling a splash screen and no others do then I'd say there is a problem

I was under the impression that points were being lost, not gained, with the splash screens disabled.

 

[Doomtrooper]

I've found 3DMark is too limited to really test between two cards capability. Reason being is that each card for one could have a drastically different Texture Level of Detail which affects the final results. I remember the GF2 with its notorious suck ass texture compression and the Radeon was putting out significantly better images, yet 3DMark was oblivious.

Here we go again...

Have you even seen a GeForce 2 run 3DMark2001SE using texture compression? It looks just fine. The "notorious suck ass texture compression" is not suck ass anymore, if it ever was. And do you have any idea what the difference is in 3DMark2001SE score between mipmap detail set to "Best Image Quality" and "Best Performance"? A whopping 25 points. And lest we forget, mipmap LOD adjusting is not specific to NVIDIA. In fact, as I recall it wasn't their idea to introduce such a thing into the world of 3D graphics cards, and they only did so in response to 3DFX's use of the technique when they were attempting to find any way possible to make their products appear to have better quality than their competitors. So if you want to blame anyone, blame 3DFX.

Yes it was suck ass, I had one and had to enable the 'hack' that basically lowers the amount of compression..kind of defeats the idea doesn't it and took as much as a 50% peformance hit when disabling it on my 32 meg GTS.

Notice the title of the link...
http://www.anandtech.com/video/showdoc.html?i=1288&p=7

 

[Doomtrooper]

I distinctly remember a thread, not in the too distant past, that had countless people arguing over whether ATI cards were rendering the game tests "properly" based on the Image Quality tests in 3DMark. Having NVIDIA cards getting a score advantage due to users being able to adjust the LOD bias (and therefore having awful looking images) isn't MadOnion's problem - that's an issue for the video card manufacturers. If ATI wanted a level playing field on that issue, then all they need to do is allow LOD bias adjustments by the user in the driver panels.

So for nobody has successfully argued that Q3A is never used as a benchmark to review video cards with and therefore assess their viability as a purchase. Magazines and websites do it alike - it is game, yes you are all correct on that, and no 3DMark isn't but they both get used to benchmark, test and compare video cards and therefore influence the views of the buying public.

A GeForce4 4600 only gets a score a small amount higher than a Radeon 8500 - or at the very least, it does in my machine - and in all the D3D games I've compared to the with:

Max Payne
Morrowind
Dungeon Seige
IL-2 Sturmovik

...the GF4 is faster than the Radeon 8500. Not excessively so but the point is, the card gets a higher score in 3DMark (about 800 points more) and the above games run faster too. It's faster in OpenGL games too but there isn't a universal OGL benchmark that both cards have a level playing field in.

If a certain brand of video card can gain approx 200-800 points by disabling a splash screen and no others do then I'd say there is a problem

I was under the impression that points were being lost, not gained, with the splash screens disabled.

Yes they are being LOST due to the screens are enabled by default, so when disabling the screens the score drops meaning.....what is being done on the splash screen that is giving 200-800 points ?? There is no 3D rendered scene to be getting any points on during the splash screen.