Late, noisy, HUGE!!! - makes 17k Mark...
- Thread starter T2k
- Start date
- Status
D3D or OGL? 8)
A sidenote on the topic:
I don't have anything against madonion/futuremark, nor their applications. The point where I oppose constantly to, is the use and resulting interpretation/conclusions either individuals or reviewers get from their applications.
Wether that single point is within the company's responsibilities to clarify or not is an entire chapter of it's own, yet in all fairness I've seen worm more than often attempting to put things into a better perspective in more than one occassions.
I'm sorry if I can't see a conspiracy in each and every corner out there (not denying that they don't exist, but not to the degree some may think); I don't see a way either to keep a rather respectable group of individuals from constant juvenile pissing contests and often even upgrade their systems solely based on 3dmark/FutureMark scores.
If you don't agree with the perception the application receives by the widest majority then do something about it, instead of blaming it simply straight on the developing company. Cut out the enthusiasm of groups like that and IHV's will end up with less sales and it will affect any other company involved in the longrun too as of course future product development and frequency. Even a small degree for that can be significant.
Ailuros said:
Actually, neither. Its a suite of tests I call "ForumMark".
It has several tests:
1) FlameMark: It measures the correllation between Doomtrooper, HellBinder and flame wars. Only certain vendors' forums seemed to score well on this one, makes one wonder eh...? The results of this were astounding and brought to light other avenues of benchmarking.
2) JoeMark: One of the avenues of benchmarking that came to light was the measurement of Joe getting the last word in. Suprisingly, every forum scores well on this one.
3) ChalnothMark: Also known as ATI-hata-Mark.
3) Nazi/FascistMark: A simple tally of accusations. Most useful for political discussion boards.
4) FanboiMark: Very similar to Nazi/FascistMark. Tuned for computer forums, however.
5) Snidemark: Because I have personal insight on this one, its usually the most accurate test of all. (One might say it was aimed at promoting a particular user....but that would be fodder for
6) ConspiracyMark: It uses a complex scatter/gather with fuzzy logic algorithm to piece together the different conspiracy theories found hidden deep within the forum Codex. It applies numerology and other cabbalistic methods for returning the encoded messages. This forum apparently returns "Weebles Wobble...". Perhaps the algorithm isn't working quite right.
[EDIT] I forgot HumorMark. It scored this particular post rather low.
I dont agree that seeing a *landslide* of circumstantial evidence, and using the massive amount of evidence to make an educated decision to be "reading things" from a given situation.
When 3dmark was first released, even NVIDIA came to the forefront to explain that their unrealistic lead in the benchmark was just that- unrealistic. 3dgpu had a nice article posted by NVIDIA concerning 3DMark and Geforce cards clearly explaining to folks this reality.
3DMark2001 came out with the Geforce3 and inhibited other videocards from running the Nature test, and docked their scores accordingly. The bottom line is- a card that might have scored a massively high score in Game 1, 2 and 3 would fall under a lesser card that can run Games 1,2,3 and Nature tests. It should also go without saying that *all* videocards can be witnessed running Nature in "demo" mode. This oversite was explained away as "well, the demo mode version of Nature isn't coded the same as the benchmark mode.. so sorry, only the Geforce3 will be allowed to run this test."
The new "SE" version then adds a PS1.4 "Advanced Shader test", but their ethic flips 180 degrees and allows a non-1.4PS to be backwards coded to use PS1.1/1.2 to emulate the same effect. Add to this how this "Advanced Shader" didn't use anything that a lesser shader (read- NVIDIA's implementation) couldnt do in a single pass fashion. There is nothing "shader" or "advanced" about this test. It's a marketing tool to draw strength away from single-pass/PS1.4 capability, from which scores from cards with the advantage were fictionally illustrated to have "no benefit" according to The Onion. This says nothing about how such a test was selectively chosen to not be part of the final score either.
It's also no secret the alleged "reference" IQ tests are included for one simple purpose- they score NV hardware better, regardless of actual IQ and should be removed entirely. Picking a reference with horrible banding, poor color accuracy and other clearly visible issues only because the refrast mode closely follows the standards of one particular IHV's choice in meeting "the minimum acceptable reference" and docking other IHVs that have exerted efforts to exceed this low standard of quality should never be called an "Image Quality" test.
It doesn't take a rocket scientist to see the clear and visible trend in the product. It is also a clear assumption that 3DMark2003 will likely contain the same kinds of strategies, but the blind will just explain it away as well.
The real question is- to what degree of evidence is needed to convince someone of the reality? Does the benchmark have to physically adjust scores with a -2000 handicap if "IHV != Desired" or what?
I heard from a friend who also knows this other guy and his dad too that:
3DMark03 was designed exclusively with CG and this was the_only_reason CG was created.
The upshot is that it will run terribly well with a GF3, GF4 and GFFX however with any other card from any other IHV (including the R300) 3DMark03 will drop to DX6 support only. Yes you heard that right, we are talking about the_real_deal here!
This is top secret information gleaned at considerable expense and breaking not one but 4 NDA's.
If you don't hear from me again - now you know why!
The Truth is Out There
3DMark03 was designed exclusively with CG and this was the_only_reason CG was created.
The upshot is that it will run terribly well with a GF3, GF4 and GFFX however with any other card from any other IHV (including the R300) 3DMark03 will drop to DX6 support only. Yes you heard that right, we are talking about the_real_deal here!
This is top secret information gleaned at considerable expense and breaking not one but 4 NDA's.
If you don't hear from me again - now you know why!
The Truth is Out There
Oh really?RussSchultz said:Speaking of which, I've got this new benchmark I'm selling....
Why not sending me a free version for evaluation? Heh.. Sure!Magnum PI said:
Well, believe it or not but I do find it amusing to read year after year the same stories everytime we release anything.Ailuros said:
RussSchultz said:5) Snidemark: Because I have personal insight on this one, its usually the most accurate test of all. (One might say it was aimed at promoting a particular user....but that would be fodder for
DAT'S ME!!!!!!
EDIT:Geez, you would think I'd remember to use the preview button a bit more.......
RussSchultz said:
You forgot one, Russ, and this isn't directed at any one person:
ScornMark--those who clearly favor a particular IHV and yet possess enough technical knowledge to avoid the appearance of your typical f@nboy. Able to heap mounds of derision upon lesser f@nboys without feeling hypocritical about it.
worm,
I'd like your input on the faults I've outlined that I find with 3dmark, here, and any information you could share about the priorities going forward. Conspirarcy theories aside for now, I find much lacking in several aspects of 3dmark. Is the goal moving forward only to make a nice looking benchmark using timely features for graphical effect, or is significant effort also being directed towards facilitating equivalency and validity of comparison? Has avoiding issues such as the Kyro issue corrected in 2001SE been prioritized?
One idea, for example, is a brute force comparison of selected image fragments from select times in a benchmark sequence, randomnly selected, to reference images based on card/settings, with tools for users to select settings for benchmarking based on their own perceived equivalency. I say brute force because if compared to a reference from the same card and settings (if those references are available to users for their own comparison), the problem of error determination can be greatly simplified (even a XOR would be meaningful, though something better should be implementable).
Such an idea, for example, would greatly shift the focus from "big number" benchmarking to something more meaningful for the gaming experience of users. Atleast, in my view it would (I'm asking if you share such a view, and details about why your view differs if it does).
What I wonder is if your team has expended effort in anything of the sort, and if not, why not. To me, the goal of "marketing a pretty demo that generates numbers for comparison" is not synonymous with the goal of "making a meaningful benchmark". That is not to say it is mutually exclusive, however...I'm trying to establish if your team has only one as a goal, or both.
I've recognized several issues with my example idea, but I've also thought of answers for each such that they do not prevent execution, and it seems to me that the more extensive tools along this line would be a fine selling point for registration. If you have your own issues about the viability of implementing such in mind, please separate that discussion from the question of this is something you and your team consider desirable, if you can can answer such at this time.
I'd like your input on the faults I've outlined that I find with 3dmark, here, and any information you could share about the priorities going forward. Conspirarcy theories aside for now, I find much lacking in several aspects of 3dmark. Is the goal moving forward only to make a nice looking benchmark using timely features for graphical effect, or is significant effort also being directed towards facilitating equivalency and validity of comparison? Has avoiding issues such as the Kyro issue corrected in 2001SE been prioritized?
One idea, for example, is a brute force comparison of selected image fragments from select times in a benchmark sequence, randomnly selected, to reference images based on card/settings, with tools for users to select settings for benchmarking based on their own perceived equivalency. I say brute force because if compared to a reference from the same card and settings (if those references are available to users for their own comparison), the problem of error determination can be greatly simplified (even a XOR would be meaningful, though something better should be implementable).
Such an idea, for example, would greatly shift the focus from "big number" benchmarking to something more meaningful for the gaming experience of users. Atleast, in my view it would (I'm asking if you share such a view, and details about why your view differs if it does).
What I wonder is if your team has expended effort in anything of the sort, and if not, why not. To me, the goal of "marketing a pretty demo that generates numbers for comparison" is not synonymous with the goal of "making a meaningful benchmark". That is not to say it is mutually exclusive, however...I'm trying to establish if your team has only one as a goal, or both.
I've recognized several issues with my example idea, but I've also thought of answers for each such that they do not prevent execution, and it seems to me that the more extensive tools along this line would be a fine selling point for registration. If you have your own issues about the viability of implementing such in mind, please separate that discussion from the question of this is something you and your team consider desirable, if you can can answer such at this time.
Hrmm.. Actually no. Game Test 4 (Nature) is a DX8 game test, while the demo is DX7. If the GT4 would have had a fallback to DX7, it wouldn't have been the same at all. For example the demo didn't have the lake. Why? Well, it simply wouldn't have been possible to do it in DX7. That alone made it impossible to have a fallback to DX7 in the Nature Test.Sharkfood said:
Oh where have you been the last year? This topic has been all over the place, but here it goes.. We didn't make the PS1.4 into the Game Tests simply because we wanted the 2001 and 2001 SE scores to be comparable. We didn't want to add 1 more Game Test that would have shifted the whole scoring system of the program. So, we made it as a feature test. Simple as that really.Sharkfood said:
I kinda missed your point here.. You mean that the reference images in 3DMark2001 / SE are .. faulty/misleading or something?Sharkfood said:
*yawn* and btw, it is 3DMark03 and not 2003.Sharkfood said:
What evidence, if I may ask? I haven't seen 1 single evidence of any conspiracy. Show me 1. I mean _real_ evidence and not some "yadda yadda" rumors..Sharkfood said:
Except for the fact that there isn't a card that can only run GT1, 2, and 3 massively better than a card that can run all 4 tests.
Capable cards
Radeon 9700/9500
Radeon 9000/9100
Radeon 8500
GeForce4 Ti
GeForce3 Ti
GeForce3
Xabres
Parhelia-512
Non-capable cards
Radeon 7500/7200/7000
Radeon
GeForce4 MX
GeForce2 GTS/Pro/Ultra/Ti
GeForce
G550/450/400
Kyro I/II
...and so on...
Perhaps the only cards that run close to each other, in terms of running Nature and non running Nature, are the GF4 MX and the 9000.
It was called the "Advanced Pixel Shader" because more was being done with pixel shaders than in the first Pixel Shader test. That simply had bump mapping from a static cube map for the water material; the APS test used reflection, refraction and Fresnel maps, plus the standard cube bump map - that's why it's "advanced". Not because it is just done with PS1.4 or 1.1 (note that Nature is actually done with 1.0 AFAIK).
"Fictionally illustrated"? That one's new to me! Where was this all done?
Score? The Image Quality tests don't "score" anything. How do you know the refrast follows one manufacturer more than another? Have you tested every single graphics card, with every driver combination possible, that can run 3DMark2001? There is nothing within the IQ tests to say that the refrast IS the best method - it is simply there as a consistent reference point, as driver setting don't change it.
And so...what? You're a great visionary? Apparently you can see the "great hidden truth" behind all of this....I'm left wondering what, if anything, this means. Should we all listen to what you have to say and follow your words "blindly"? I'll say this again...it's oh-so easy for everyone to hide behind the defensive anonymity that is the WWW and jeer and protest. It doesn't seem to matter who or what the target is - Intel, AMD, NVIDIA, ATI, Carmack, Hook, Futuremark, Mrs Miggins - there's always seems to be somebody that will fervently have a go at something/someone; however, these people never seem to be the ones actually doing anything about the "problems".
From the evidence of the numerous posts on these boards, there seems to be enough talent/experience/ego to make a D3D to challenge 3DMark. If it is such a great unwashed evil that is the epitome of capitalism, then why not produce something to save us all?
Edit: Looks like Nicke beat me to it!
On a side note-
I don't think this is the fault of 3DMark or Futuremark, but instead the way ATI reports the device identification string with their drivers.
Whether you have a 9500, 9500 Pro, 9700, 9700 pro, the ATI driver identification string is "Radeon 9700/9500 Series"
I believe this is just a vanilla string sent from the drivers from which MadOnion just uses as the key for category.
This was also an issue back when 3dfx was messing around with this string between different driver revisions. Depending on which month it was, the drivers would be categorized as "3dfx Voodoo Series", "Voodoo 4/5", "3dfx Voodoo 5" etc.etc. And people making home-brewed drivers would hack the ASCII and create more flavors for the same product.
Maybe DEV ID or similar unique indentifier should be adopted if there is no standard for how IHV driver manufacturers set this string based on hardware detected.
I don't think this is the fault of 3DMark or Futuremark, but instead the way ATI reports the device identification string with their drivers.
Whether you have a 9500, 9500 Pro, 9700, 9700 pro, the ATI driver identification string is "Radeon 9700/9500 Series"
I believe this is just a vanilla string sent from the drivers from which MadOnion just uses as the key for category.
This was also an issue back when 3dfx was messing around with this string between different driver revisions. Depending on which month it was, the drivers would be categorized as "3dfx Voodoo Series", "Voodoo 4/5", "3dfx Voodoo 5" etc.etc. And people making home-brewed drivers would hack the ASCII and create more flavors for the same product.
Maybe DEV ID or similar unique indentifier should be adopted if there is no standard for how IHV driver manufacturers set this string based on hardware detected.
Niklas:
"What evidence, if I may ask? I haven't seen 1 single evidence of any conspiracy. Show me 1. I mean _real_ evidence and not some "yadda yadda" rumors.."
Here's the great thing about paranoid people, they ALWAYS have a logical comeback to reply with to something like that. This is what they'd say:
"The fact the evidence hasn't been found doesn't mean it isn't there! That's why conspiracies are called conspiracies!"
So what would you reply to that, huh?
Then there's always that hardware analyzer thingy that Nvidia helped you guys with who recommended people to go get slower video cards from the preferred manufacturer to replace the superior hardware in their systems... Guess that's all the evidence some people need. Well actually, I don't know how you Onion guys would explain that one in a satisfactory manner, so in this case I guess any conspiracy theorist would be correct.
*G*
"What evidence, if I may ask? I haven't seen 1 single evidence of any conspiracy. Show me 1. I mean _real_ evidence and not some "yadda yadda" rumors.."
Here's the great thing about paranoid people, they ALWAYS have a logical comeback to reply with to something like that. This is what they'd say:
"The fact the evidence hasn't been found doesn't mean it isn't there! That's why conspiracies are called conspiracies!"
So what would you reply to that, huh?
Then there's always that hardware analyzer thingy that Nvidia helped you guys with who recommended people to go get slower video cards from the preferred manufacturer to replace the superior hardware in their systems... Guess that's all the evidence some people need. Well actually, I don't know how you Onion guys would explain that one in a satisfactory manner, so in this case I guess any conspiracy theorist would be correct.
*G*
I got to admit that the Performance Analyzer mentioned above was pretty damning.. and funny
However I suppose you cannot blame Futuremark/Madonion/Futuremark for looking at exploiting their products to generate revenue from other sources.
The thing to remember about 3DMarkxx is that it is a free download and you dont have to pay for anything to run it, compare your system with another and use it to make sure your system is stable. Even now reviewers are slowly moving away from synthetic benchmarks in favour of real application/game banchmarks (and it has taken too long for this to occur at any rate).
However I suppose you cannot blame Futuremark/Madonion/Futuremark for looking at exploiting their products to generate revenue from other sources.
The thing to remember about 3DMarkxx is that it is a free download and you dont have to pay for anything to run it, compare your system with another and use it to make sure your system is stable. Even now reviewers are slowly moving away from synthetic benchmarks in favour of real application/game banchmarks (and it has taken too long for this to occur at any rate).
It's not cheating to reduce the quality of a specific game in order to give it better FPS? You learn new things everyday!Magnum PI said:
http://www.3dcenter.de/artikel/2001/10-24_a.php
http://www.hardocp.com/files/cool_stuff/quackcompare.zip
Slides, please, this has been discussed ad-nausea numerous times (as has this whole (yawn) 3Dmark thing). WRT to the Quake string, that could be seen in the ATI drivers long before 8500 was released, and game specific optimisation does occur, with everyone, whether we like it or not. There are many, many ways of detecting a game and optimising for it in the drivers (detecting the vertex format the game uses can be one for instance), the string initialisation is just a rather rudimentry one.
Oh, and please stop wasting database space with unnecessary nested quotes.
Oh, and please stop wasting database space with unnecessary nested quotes.
Sorry about the quotes, but do you see an image difference? Should any company be allowed to reduce quality for a specific game in order to increase fps? Is reducing image quality, "optimizing"?
http://www.hardocp.com/files/cool_stuff/quackcompare.zip
http://www.hardocp.com/files/cool_stuff/quackcompare.zip
- Status