Late, noisy, HUGE!!! - makes 17k Mark...
- Thread starter T2k
- Start date
- Status
Himself said:
Unfortunately, code is code. You can target any performance result you so choose depending upon the showcase you build.
For example, you might make a scene with a fairly simple shader (one that gains little benefit from PS1.4) yet surrounding scene detail requires 6+ texture layers as to require multiple passes. In this case, the additional passes needed by the shader are skewed by the larger impact the texture passes require. There are literally thousands of ways to skew results.
This is the reason for dissention with the Advanced shader test. It truly does not flex the muscle of PS1.4, yet is a new test added to add comparison between PS1.4 on one IHV to be compared to PS1.1 on another IHV. The results are baffling- especially given the ability to test PS1.1 and PS1.4 on the same IHV and see little to no change either.
It would have made more logical sense to recode Nature to use the highest shader version possible... Or if concerns of scores are truly involved, then do this to the existing ocean pixel shader test. Why set out to create an entirely new benchmark test and advertise it heavily with the buzzword "PS1.4" all over the readme/help files/website unless there was some message intended to be delivered in this new, ground-up, test? And the results that can be perceived from the test also speak volumes as they are baffling.
Hi Sharkfood,
The Readme doesn't mention PS1.4 at all.
The help files contain this only:
"Advanced Pixel Shader
This is a new test included in 3DMark2001 SE, and it uses Pixel Shader version 1.4, introduced in DirectX 8.1. The same effect can also be achieved using Pixel Shader 1.0, but then rendering the water surface requires two passes. Graphics hardware that supports Pixel Shader 1.4 (or higher) render the water in a single pass. The water surface in this test is more realistic than in the test above. Instead of just an environment bump map reflecting a cube map, this test uses a ripple texture (traditional environment bump map), a reflection texture, a refraction texture and a Fresnel texture. The resulting effect is a water surface that has ripples from for example the wind, shows a rippled reflection and a rippled refraction (view of objects beneath the water surface) + the reflection intensity is adjusted according to the camera's angle to the water surface. For example looking perpendicularly at the water surface will show only refracted light rays, or what's under the water surface, and looking almost parallel to the water surface will show the reflection only."
The web site states this:
"A new DirectX8.1 feature test showing Vertex Shaders & Pixel Shaders 1.4."
That's all. I don't see any heavy advertising. As worm has stressed multiple times (and Neeyik as well, IIRC), "advanced pixel shader" means nothing other than that THIS shader test uses more complicated shaders than the previous ones. That's all. Hardly any stress on PS1.4 at all. And if you call that one web site quote "heavy advertising," I'm afraid you need a reality check.
ta,
-Sascha.rb
Err, can you show us the "heavy advertising" you mention?Sharkfood said:
The Readme doesn't mention PS1.4 at all.
The help files contain this only:
"Advanced Pixel Shader
This is a new test included in 3DMark2001 SE, and it uses Pixel Shader version 1.4, introduced in DirectX 8.1. The same effect can also be achieved using Pixel Shader 1.0, but then rendering the water surface requires two passes. Graphics hardware that supports Pixel Shader 1.4 (or higher) render the water in a single pass. The water surface in this test is more realistic than in the test above. Instead of just an environment bump map reflecting a cube map, this test uses a ripple texture (traditional environment bump map), a reflection texture, a refraction texture and a Fresnel texture. The resulting effect is a water surface that has ripples from for example the wind, shows a rippled reflection and a rippled refraction (view of objects beneath the water surface) + the reflection intensity is adjusted according to the camera's angle to the water surface. For example looking perpendicularly at the water surface will show only refracted light rays, or what's under the water surface, and looking almost parallel to the water surface will show the reflection only."
The web site states this:
"A new DirectX8.1 feature test showing Vertex Shaders & Pixel Shaders 1.4."
That's all. I don't see any heavy advertising. As worm has stressed multiple times (and Neeyik as well, IIRC), "advanced pixel shader" means nothing other than that THIS shader test uses more complicated shaders than the previous ones. That's all. Hardly any stress on PS1.4 at all. And if you call that one web site quote "heavy advertising," I'm afraid you need a reality check.
ta,
-Sascha.rb
nggalai-
You provided it-
"This is a new test included in 3DMark2001 SE, and it uses Pixel Shader version 1.4"
and
"A new DirectX8.1 feature test showing Vertex Shaders & Pixel Shaders 1.4."
I don't know how much clearer you need for a test being advertised as a PS1.4 benchmark.
You provided it-
"This is a new test included in 3DMark2001 SE, and it uses Pixel Shader version 1.4"
and
"A new DirectX8.1 feature test showing Vertex Shaders & Pixel Shaders 1.4."
I don't know how much clearer you need for a test being advertised as a PS1.4 benchmark.
Hi Sharkfood,
I still don't get it. You call two lines of text among lots and lots of content on a web site, readme file, help files, and press release "heavy advertising?" These two lines weren't even put in prominent spaces--just as part of a list or regular content. No highlighting, either.
And where is it advertised as a PS1.4 benchmark? It's advertised as a feature test that can take advantage of PS1.4. It does--on PS1.4, it takes one pass, on earlier hardware two. Where's my problem understanding you?
ta,
-Sascha.rb
I still don't get it. You call two lines of text among lots and lots of content on a web site, readme file, help files, and press release "heavy advertising?" These two lines weren't even put in prominent spaces--just as part of a list or regular content. No highlighting, either.
And where is it advertised as a PS1.4 benchmark? It's advertised as a feature test that can take advantage of PS1.4. It does--on PS1.4, it takes one pass, on earlier hardware two. Where's my problem understanding you?
ta,
-Sascha.rb
Well, if you dont see the references to PS1.4 in every single mentioning of the advanced shader test, I don't know what else to say.
I guess it is true some people only see what they want to see. What I see is every single mentioning of the test (as provided by yourself) as making it QUITE clear the benchmark uses PS1.4 (highlighted in bold from your quotes.. still not seeing it?).
I'm not understanding the disconnect here. The APS is documented, advertised, and clearly shown in all references as having PS1.4. I'm not sure where the misunderstanding is that this somehow doesnt showcase the test as being one for PS1.4...
I guess it is true some people only see what they want to see. What I see is every single mentioning of the test (as provided by yourself) as making it QUITE clear the benchmark uses PS1.4 (highlighted in bold from your quotes.. still not seeing it?).
I'm not understanding the disconnect here. The APS is documented, advertised, and clearly shown in all references as having PS1.4. I'm not sure where the misunderstanding is that this somehow doesnt showcase the test as being one for PS1.4...
Babel-17
Veteran
Lol, basically, the only benchmarks that concern me are based on UT2k3. I'm a fan of the game itself and I expect to buy every single title using the game engine or variants thereof. While I do have a R9700 I am never the less not holding my breath waiting for DX9 games. I've recently had a V5 5500, a GF3 and a GF4 Ti 4600. I don't mind (well, not too much) supporting advanced features in my hardware that are a ways from being common in games but my main reason for buying a card is what it will do for me now and a maximum of 18 months (actually it's working out to less, .... the flesh is weak) down the road. When Doom 3 engined games are scheduled to hit the shelves I maybe will care about how my card handles those effects.
I can navigate my way around the internet but it's not easy finding the information I deem important in making a purchase. Benchmark this and benchmark that, I'm looking for smooth driver installs and upgrades along with game compatability (which is one of the reasons why I bought 3Dfx and then, later, nVidia) and outstanding default IQ as well as good tools to improve it further (aa and aniso). I've been monitoring multiple sites regarding ATI's status and just recently decided they finally met my criteria. No offence to 3DMark2k1, it can be a somewhat useful tool, but it played virtually no role in my decision making process.
Real games with real benchmarking tools are the future. For info on IQ I count on boards like here and slackercentral (there are others). I read the threads on many other gaming boards and I get sad. For the sake of not seeing the few good message boards out there go up in flame wars I hope the next iteration from Mad Onion uses an engine that will allow cards to showcase abilities that are meaningful to people who play games. By that I mean current games, newly released games, and games that really are in the pipeline.
Scenes that are "posterboys" for the need of good antialiasing and anisotropic filtering are a "must" imo for IQ comparisons to be of any real value.
I can navigate my way around the internet but it's not easy finding the information I deem important in making a purchase. Benchmark this and benchmark that, I'm looking for smooth driver installs and upgrades along with game compatability (which is one of the reasons why I bought 3Dfx and then, later, nVidia) and outstanding default IQ as well as good tools to improve it further (aa and aniso). I've been monitoring multiple sites regarding ATI's status and just recently decided they finally met my criteria. No offence to 3DMark2k1, it can be a somewhat useful tool, but it played virtually no role in my decision making process.
Real games with real benchmarking tools are the future. For info on IQ I count on boards like here and slackercentral (there are others). I read the threads on many other gaming boards and I get sad. For the sake of not seeing the few good message boards out there go up in flame wars I hope the next iteration from Mad Onion uses an engine that will allow cards to showcase abilities that are meaningful to people who play games. By that I mean current games, newly released games, and games that really are in the pipeline.
Scenes that are "posterboys" for the need of good antialiasing and anisotropic filtering are a "must" imo for IQ comparisons to be of any real value.
We care more than you think, and I'm not even a part if the "Database team". It is our lifeline to stay unbiased. That should be clear to you by now..Doomtrooper said:
FYI, it's Futuremark nowdays, and not MadOnion.com.
Sharkfood,
You must be kidding, right? 3DMark2001 SE was (and still is) advertised/marketed as a "DirectX8.1 Benchmark". Not "PixelShader 1.4 Benchmark". The Advanced PixelShader Test is not a "PixelShader 1.4 Test". It is an Advanced PS Test. The test uses PS1.4 if available in the HW. Is that so hard to comprehend?
Well, you said it.Sharkfood said:
Babel,
That lies more or less within the point I tried to make (I don't recall even how many pages ago LOL). It depends how somebody uses/interprets an application/game.
In that case even a game can fall in the same category as FutureMark's applications, where a user overclocks/tweaks/upgrades etc solely based on XYZ score whereby it's doubtful that he actually ever plays it (I don't mean you heh).
I agree on all of your points; I don't deny using any of the popular applications/benchmarks, yet I do not only use a far wider facette of games than those to reach a conclusion, but I also try to fit my needs within a fine (call it subjective if you want) balance between IQ and performance. Xabre perfoms fine for it's price too
Finally if I decide to benchmark something then it usually includes the ingame settings and resolution/colour depth I use while gaming too. Those I want to tweak/optimize for and not senseless 20K that or 300fps this score in whatever.
That lies more or less within the point I tried to make (I don't recall even how many pages ago LOL). It depends how somebody uses/interprets an application/game.
In that case even a game can fall in the same category as FutureMark's applications, where a user overclocks/tweaks/upgrades etc solely based on XYZ score whereby it's doubtful that he actually ever plays it (I don't mean you heh).
I agree on all of your points; I don't deny using any of the popular applications/benchmarks, yet I do not only use a far wider facette of games than those to reach a conclusion, but I also try to fit my needs within a fine (call it subjective if you want) balance between IQ and performance. Xabre perfoms fine for it's price too
Finally if I decide to benchmark something then it usually includes the ingame settings and resolution/colour depth I use while gaming too. Those I want to tweak/optimize for and not senseless 20K that or 300fps this score in whatever.
Hi Sharkfood,
Oh, I see the references to PS1.4 clearly. But I still don't understand you. Do you really call that "heavy advertising?" The average joe who downloads the application or installs it from a magazine CD will never even read anything about PS1.4. (How many people read the readme?) We're talking about some very short pits of texts that mention PS1.4, quotes that also are absolutely clear about how PS1.4 are used in the advanced pixel shader feature test. It even explains the advantages of using PS1.4 in the APS test. Where's the problem?
Did ATI "heavily advertise" its presampled displacement mapping? It is mentioned in the papers, after all.
Is your claim that this is "false advertising," then? That the APS test doesn't use PS1.4 to its fullest extend, but still claims to run on PS1.4 (which it does) and hence favours certain IHV hardware and skews the user's perspective? In that case, as nAo suggested, somebody should please provide us with a sample of a PS1.4 effect or better still, a PS1.4 "benchmark" that will show a better way to do it. The APS test already needs two passes on non-PS1.4 hardware as opposed to one on Radeon, what more do you want? An artificial scenario that strongly favours PS1.4 coding, even though showcasing PS1.4 wasn't the intend of this test as is clearly shown in a) the LACK of "heavy advertising" of PS1.4 and b) the explanations by Futuremark as seen in this and earlier threads?
Well, for a more constructive posting: a suggestion to the Futuremark people who might still lurk in this thread.
I really would prefer shader tests to run similarly to the bump mapping feature tests in 3DMark2000. e.g. have PS1.1, 1.4, and 2.0 feature tests, check if the card supports the feature and if no,t don't run the test ("not supported"). This way, the results from different architectures would be somewhat more comparable. As is with the APS test in 3DMark2001, the results are pretty much useless as different code paths are used for different hardware.
No offence,
-Sascha.rb
Oh, I see the references to PS1.4 clearly. But I still don't understand you. Do you really call that "heavy advertising?" The average joe who downloads the application or installs it from a magazine CD will never even read anything about PS1.4. (How many people read the readme?) We're talking about some very short pits of texts that mention PS1.4, quotes that also are absolutely clear about how PS1.4 are used in the advanced pixel shader feature test. It even explains the advantages of using PS1.4 in the APS test. Where's the problem?
Did ATI "heavily advertise" its presampled displacement mapping? It is mentioned in the papers, after all.
Is your claim that this is "false advertising," then? That the APS test doesn't use PS1.4 to its fullest extend, but still claims to run on PS1.4 (which it does) and hence favours certain IHV hardware and skews the user's perspective? In that case, as nAo suggested, somebody should please provide us with a sample of a PS1.4 effect or better still, a PS1.4 "benchmark" that will show a better way to do it. The APS test already needs two passes on non-PS1.4 hardware as opposed to one on Radeon, what more do you want? An artificial scenario that strongly favours PS1.4 coding, even though showcasing PS1.4 wasn't the intend of this test as is clearly shown in a) the LACK of "heavy advertising" of PS1.4 and b) the explanations by Futuremark as seen in this and earlier threads?
Well, for a more constructive posting: a suggestion to the Futuremark people who might still lurk in this thread.
I really would prefer shader tests to run similarly to the bump mapping feature tests in 3DMark2000. e.g. have PS1.1, 1.4, and 2.0 feature tests, check if the card supports the feature and if no,t don't run the test ("not supported"). This way, the results from different architectures would be somewhat more comparable. As is with the APS test in 3DMark2001, the results are pretty much useless as different code paths are used for different hardware.
No offence,
-Sascha.rb
Typedef Enum
Regular
No doubt...
A few sparse words in some Readme text file, that virtually nobody ever reads, doesn't comes close to heavy advertising.
If it were splashed all over the application dialog...presented within the benchmark...maybe. But you're really fighting an uphill battle if you're trying to insist that the mentioning of PS 1.4 in some text file that nobody reads is heavy advertising.
A few sparse words in some Readme text file, that virtually nobody ever reads, doesn't comes close to heavy advertising.
If it were splashed all over the application dialog...presented within the benchmark...maybe. But you're really fighting an uphill battle if you're trying to insist that the mentioning of PS 1.4 in some text file that nobody reads is heavy advertising.
That's the whole point. It was next to impossible to download or get on a CD without reference to the new test, which was clearly identified as a PS1.4 test (JUST like the two examples you have pulled up).
Seriously, you guys are acting like the Advanced Pixel Shader's only mention to PS1.4 was on display at the offices, in the basement, without stairs and behind a door that say "Beware of the leopard." Sorry, try again.
I guess if I went to forums around the globe and posted "Hey guys, I just discovered the APS has PS1.4!" the result would be a bunch of people "Wow! I never knew that!" Give it a rest.
There is the point of disagreement. The results on a Radeon already disprove this (just disable PS1.4). The "point" of the test is pretty clear- that PS1.4 performs similar or worse than PS1.1 (as illustrated by scores or any amount of unbiased testing) and showcases this message nicely.
I'm okay with that. Really. It just goes hand in hand with my prediction of how this will not be the case with 3dmark03. I guarantee it. Any DX9.0 feature tests will be used as showcases (once again) to make the same kinds of fictional statements about performance. The same kinds 3dmark99, 2000, 2001 and 2001SE have flip-flopped concerning levels of complexity, static/prepathed HW T&L, excruciatingly specialized texturing and a PS1.4 test that gains little to no performance between it's runtime performance on PS1.0/1.1.
I await 3dmark03 to either prove or disprove this theory.
It all comes down to a design approach to a benchmark. If all IHV's are tossed out and the focus becomes one of an API, the end result tests will be substantially different. The moment coding becomes custom tailored to one or more IHV's is the moment the usefulness of said tool diminishes and at the same moment becomes nothing more than another marketing tool.
Like Democoder put- to test shader performance, take something generic and not tied to an IHV's capabilities. Take something non-realtime in Renderman or similar and create the highest level complexity the API supports and write the most optimum methods (by API, not by IHV) for lower/fallbacks then let it rip. The only code changes from that point forward would be for possible bugs.. not optimizations.
You dont create Utopian scenarios in a benchmark for any particular featureset.
My grip is how poorly they handled the K2. There was a well know bug that effected the performance of the K2. It was almost 6 months before a patch was released. I understand you can not go releasing patches with every little issue or bug. And I know full well that patches takes a lot of time to test and qualify. Still there is no excuse why they let this issues go on so long.
Selling a benchmark that you know has become the defacto standard makes you have a moral obligation to make that benchmark as accurate and hopefully as fair as possible. You also know that many peoples buying decisions are based off benchmarks such as yours. Thus if you find your benchmark is not accurately scoring a certain card then its up to you to fix it with all possible speed. 6 months is not all possible speed. You could have realsed a hot fix for the K2, but instead decieded to wait and bundle it with the other updates. Piss poor judgement.
Then while this issue was going on you marketed the performance analyzer that reads your data bases. Which in itself is fine/great. However when you know that is giving in-accurate information thanks to the kryo2 bug then why did you allow it to be marketed? Shouldn't you have waiting until the bug was fixed and then allow its use? Was it give out fair info. Heck no.
Weather or not there is something "evil" going on with 3dmark (which I dont think so) we dont now nor do I care. However there is no way you guys handled the k2 stuff fairly and accuratly. You shafted it big time. I dont think you folks woke up and say hay lets shaft the K2. It just sort of happend. And worm I understand none of this had anything to do with you.
Selling a benchmark that you know has become the defacto standard makes you have a moral obligation to make that benchmark as accurate and hopefully as fair as possible. You also know that many peoples buying decisions are based off benchmarks such as yours. Thus if you find your benchmark is not accurately scoring a certain card then its up to you to fix it with all possible speed. 6 months is not all possible speed. You could have realsed a hot fix for the K2, but instead decieded to wait and bundle it with the other updates. Piss poor judgement.
Then while this issue was going on you marketed the performance analyzer that reads your data bases. Which in itself is fine/great. However when you know that is giving in-accurate information thanks to the kryo2 bug then why did you allow it to be marketed? Shouldn't you have waiting until the bug was fixed and then allow its use? Was it give out fair info. Heck no.
Weather or not there is something "evil" going on with 3dmark (which I dont think so) we dont now nor do I care. However there is no way you guys handled the k2 stuff fairly and accuratly. You shafted it big time. I dont think you folks woke up and say hay lets shaft the K2. It just sort of happend. And worm I understand none of this had anything to do with you.
For a lot of people, they couldn't even view the nature test for a year or more until they upgraded their hardware, the advanced nature test was just another test skipped. I'm sure the SE version of 3dmark2001 was considered by many to just be a bug fix release since they couldn't see anything different. That pretty much describes me, actually, I never paid attention to the advanced shader test and never cared what version of geek speak was being used behind the curtain.
Waiter..?
Waiter..?
The 3dmark database is useless anyway.
I remember when GF3Ti200 came out it was ranked higher than the GF3-classic.
Now I know there's an easy explanation for this - GF3Ti200 was faster with the DetXP drivers than the GF3-classic with the old ones.
But it still makes the database useless.
I remember when GF3Ti200 came out it was ranked higher than the GF3-classic.
Now I know there's an easy explanation for this - GF3Ti200 was faster with the DetXP drivers than the GF3-classic with the old ones.
But it still makes the database useless.
- Status