What do you think of glexcess and friends? I think they are utter rubblish myself, I can't keep awake while they are running and the results are nigh well incomprehensible. We need more benchmarks as convenient and as well done as 3dmark, not less. The more the merrier. Wonder if the next 3dmark will have an opengl portion, that would be cool.
Late, noisy, HUGE!!! - makes 17k Mark...
- Thread starter T2k
- Start date
- Status
Doomtrooper
Veteran
Try some of these demos:
http://www.vertice.fr/Pages/demosaccueil.htm
Oh and another note..there should be no way to allow LOD bias changes, something that is laughable on a benchmark that is supposed to be testing graphics.
http://www.vertice.fr/Pages/demosaccueil.htm
Oh and another note..there should be no way to allow LOD bias changes, something that is laughable on a benchmark that is supposed to be testing graphics.
Doomtrooper said:
The problem is not isolated to 3DMark.
I'm not aware of any gfx-benchmark used on the web that checks for correctness. Not trivial to do but not impossible either. For instance, you could stash away sample frames, and compare relative to a reference, and highlight those which differ beyond what could be considered normal. At any rate, the problem is manageable, it's just that nobody bothers to do it.
3DMark has other problems as a benchmark. One which has been discussed several times here is that since it's so dominant, it makes perfect sense for manufacturers not only to optimize for it, but to actually detect it and make special cases for 3DMark. Which is not disallowed by FutureMark, but it certainly makes the results pretty useless for predictive purposes.
And of course FutureMark is a privately held corporation, and is free to have all kinds of dealings with anyone on any terms they want. That BAPCo logo on their site always made me ill at ease, but noone says a deal has to be made official like that either. We have no way of knowing what goes on inside that corporation. Only that its ultimate goal is to make money for its owners.
All benchmarks have the problem of result transferability - can the results predict performance on other problems with useful precision? But 3DMark compounds this by trying to be forward looking, so it doesn't even try to be a good descriptor of current games. Instead it makes assumptions about future directions which most probably won't be all that accurate overall, and this obviously doesn't help its validity as a general predictive tool.
Always liked the actual product though. For instance the Nature scene where they broke away from a static environment had the potential to influence the industry in a positive way. Having an obvious optimisation target like 3DMark also spurs driver performance not only for the special case but also generally, and this effect is clearly good.
But to have faith in it as a predictive benchmark?
That would be quite naive.
Entropy
demalion said:
I'm sorry to say but the truth is the Kyro ran like total crap in many other games. It should have been faster than the MX, but it was ignored by many developers and thus performed poorly. Also the lack of a TnL unit killed it, because although it was a faster "straight up" renderer than the MX, it had no TnL and when TnL games started hitting the market it died a quick death. So, 3DMark isn't as inaccurate as you might think.
I have no idea about the Parhelia, but I bet many of those Ti200s overclocked to Ti500 speed. Meanwhile I guess the Parhelia didn't overclock much if at all. Stock, the Parhelia is faster so maybe that isn't completely accurate, but even so I'd probably recommend the Ti200 over it anyway.
I guess my point is, for all that 3DMark is supposedly so inaccurate, it's results list is identical to how I'd rank the cards on the market today. Although I'd just leave the Parhelia off completely, because it is overpriced by an insane amount. Love it or hate it, all IHVs can optimize their drivers for 3DMark, so it's a level playing field. IHVs also optimize their drivers for games too, so I don't see why optimizing makes a comparison invalid. Honestly, both ATI and Nvidia have optimizations for Quake3...GOOD, it actually runs better!
http://www.nordichardware.se/artiklar/Intervjuer/English/nVidia
I have more Q&A on it's way to my inbox since the nVidia staff there couldn't answer all of my questions
Actually I didn't think the fan was very noisy at all. (Granted it wasn't exactly a quite place )
The exhaust air was pretty hot though.
Dawn was pretty choppy even without any form of AA/AF nor super high res nor any crazy polygon counts
BTW when asked about 3D mark scores by 64bits.se nV replied that you won't see anyone breaking 20k with a 3.06 GHz P4 when the stuff is running at stock speed
not that it's directly related to the subject but we also did an interview with ATi:
http://www.nordichardware.se/artiklar/Intervjuer/English/ATi
I have more Q&A on it's way to my inbox since the nVidia staff there couldn't answer all of my questions
Actually I didn't think the fan was very noisy at all. (Granted it wasn't exactly a quite place
)The exhaust air was pretty hot though.
Dawn was pretty choppy even without any form of AA/AF nor super high res nor any crazy polygon counts
BTW when asked about 3D mark scores by 64bits.se nV replied that you won't see anyone breaking 20k with a 3.06 GHz P4 when the stuff is running at stock speed
not that it's directly related to the subject but we also did an interview with ATi:
http://www.nordichardware.se/artiklar/Intervjuer/English/ATi
T2k said:For the records:
Of course, Parhelia IS faster in today's games than a GF3. (4x4 architecture, you know...
Are you sure? The Parhelia was trounced pretty handily by the GeForce4. Remember that the Parhelia has no memory bandwidth savings techniques or occlusion detection, while the GeForce3 does.
Doomtrooper
Veteran
It was only trounced when nothing was enabled, when FSAA and AF was enabled in some high resolution benchmarks it outperformed the Ti series by a good margin...in fact I noticed this from the very 1st reviews but as usual the 1st thing people did was talk about how low the 3Dmark scores was then proceeded to show their $400 T4600 benchmarks running with nothing on at all
Nagorak said:
You might ask yourself why "its results list is identical to how I'd rank the cards on the market today". It's a remarkable coincidense given how small the percentage differences are, wouldn't you say? Since you can hardly have owned and tested all current cards, what are you basing your judgement of speed on, really? Hmm. Could it be that your perception of card speed has been influenced by the very ... nah.
Introspection is not for everyone.
3DMark has mapped OK to general game performance but the reason is simple - apart from host effects, the scores have always been largely a measure of fillrate.
Entropy
Just my opinion.
I did, wasn't impressed, I'm sure there is something 3D folks will find facinating and impressive technically there somewhere. Novamark especially gets my yawn award, to me is an array of trees and some simple terrain rendering with some acne trees.
Is there anything in the code that "allows" LOD bias changes? I don't think that there is much that a benchmark can do to prevent things like that, all it can do is send commands to an API, it has no control over what the driver does to implement it. A driver could be null renderer if it wanted to, or only render every other frame, nothing a benchmark can do about that. What I guess you mean is that there should be some form of image standard that the output is measured against. I think that any use of FSAA or anisotropic would make that next to impossible, one's card's visual optimization is someone else's incorrect result. Best you could do really is save some screenshots, but that would make database storage ala 3dmark, pretty much impractical.
I think people overestimate what lengths video card companies are willing to go to in order to cheat at benchmarks, if they can isolate bottlenecks and improve code, sure, they optimize that way, but outright trying to cheat, that's reaching. I don't really buy that level of stupidity, it's a very well known sequence of image coming out of the thing, any variance and it will get noticed eventually. Sure, optimizing for 3dmark2001 won't get your old version of deer hunter to zip along so in that respect it doesn't track games, but that isn't the fault of the benchmark. If you wanted a benchmark for current games, try 3dmark2000 or 3dmark99 for that matter, just up the resolution or something, games are bottom feeders when it comes to video card features or new APIs.
Nagorak,
I agree about the KyroII, great in serious sam so long as you picked the right settings, FSAA or aniso, not both for example, but in just about everything else it was a binary choice between great image quality and no speed or speed and crap image quality. When things worked, good luck getting something as simple as the sims to play properly. (Wasn't my system.)
Doomtrooper said:
I did, wasn't impressed, I'm sure there is something 3D folks will find facinating and impressive technically there somewhere. Novamark especially gets my yawn award, to me is an array of trees and some simple terrain rendering with some acne trees.
Is there anything in the code that "allows" LOD bias changes? I don't think that there is much that a benchmark can do to prevent things like that, all it can do is send commands to an API, it has no control over what the driver does to implement it. A driver could be null renderer if it wanted to, or only render every other frame, nothing a benchmark can do about that. What I guess you mean is that there should be some form of image standard that the output is measured against. I think that any use of FSAA or anisotropic would make that next to impossible, one's card's visual optimization is someone else's incorrect result. Best you could do really is save some screenshots, but that would make database storage ala 3dmark, pretty much impractical.
I think people overestimate what lengths video card companies are willing to go to in order to cheat at benchmarks, if they can isolate bottlenecks and improve code, sure, they optimize that way, but outright trying to cheat, that's reaching. I don't really buy that level of stupidity, it's a very well known sequence of image coming out of the thing, any variance and it will get noticed eventually. Sure, optimizing for 3dmark2001 won't get your old version of deer hunter to zip along so in that respect it doesn't track games, but that isn't the fault of the benchmark. If you wanted a benchmark for current games, try 3dmark2000 or 3dmark99 for that matter, just up the resolution or something, games are bottom feeders when it comes to video card features or new APIs.
Nagorak,
I agree about the KyroII, great in serious sam so long as you picked the right settings, FSAA or aniso, not both for example, but in just about everything else it was a binary choice between great image quality and no speed or speed and crap image quality. When things worked, good luck getting something as simple as the sims to play properly. (Wasn't my system.
)Doomtrooper said:It was only trounced when nothing was enabled, when FSAA and AF was enabled in some high resolution benchmarks it outperformed the Ti series by a good margin...in fact I noticed this from the very 1st reviews but as usual the 1st thing people did was talk about how low the 3Dmark scores was then proceeded to show their $400 T4600 benchmarks running with nothing on at all
Only problem is, that's with FAA enabled, which is buggy. I'd take comprehensive 2x AA any day over 16x FAA that misses some edges.
And the Matrox Parhelia only supports 2-degree anisotropic, so it shouldn't be compared to the GeForce4 at 8-degree for performance.
LeStoffer said:
I think looking at all the individual scores shows pretty well what is going on with someone else's system, the fsb is listed, the cpu type and speed, the motherboard brand, etc. Quite a few people add comments on anything specific they did with their system as well, and then there are the forums. I don't think you can or want to isolate video card performance from the rest of the system artificially. A single result doesn't tell all no, but there are just about infinite combinations of results to browse through, you won't get an exact measurement but you will definitely get a ballpark idea of what does the job.
Chalnoth said:
So do you prefer SS as well?
I seem to recall you arguing quite clearly for MS.
Strange how your logic changes.
By this comment, i'd figure that you'd much rather have comprehensive 2x SSAA over MSAA that misses alpha textures and texture aliasing.
Doomtrooper
Veteran
There is a 'LOD bias hack' for 3Dmark has been for a long time.
Nagorak,
What you describe is running a popularity contest, not benchmarking. I would like to see the benchmarks of the many games where it "ran like total crap" for reference, though, to see how well the 3dmark problems reflected that.
Himself,
Strange how your criteria, which you've also stated before this, for a good benchmark is solely based on the entertainment value. Or maybe not so strange, it seems Futuremark has a similar set of priorities.
Can you see how little sense that chain of logic makes in a discussion about the validity of a benchmark? Again, this is benchmarking, not a popularity contest. 3dmark does not measure the cost of the card as part of its testing suite.
Entropy's response to Nagorak fits here so well, I'll refer you to it.
No, it is an optimization contest, not a level playing field. The problem is it is an application specific optimization contest, not a general driver optimization contest. Do we have to go over every time relative 3dmark performance of products compared to game performance has indicated this problem? Or is the jump in 3dmark performance for the GF4 that "happened" to occur around 9700 release, for example, represented in games? (An honest question, I don't own a GF4, but comments have led me to believe otherwise).
The only hope I have for 3dmark 2003 or whatever is that with the right mix of shaders and with aids to image quality comparison, optimizing for it is actually likely to benefit games in general.
Since the assertion is "invalid as an indicator of relative performance", I suggest you complete the sentence with that phrase and see if it still fits what you are trying to say. To me, when I do that it reads as something indefensible.
Are you deliberately missing the point, or you just don't care if new driver releases are driven by the goal of optimizing for a benchmark you sit and watch instead of games you play? You seem to enjoy the benchmark a lot, so perhaps you just don't. I hope you'll note my comments don't attempt to tackle Quake III...or atleast they don't here, I've made comments about that elsewhere before. For the purposes of this discussion, I'll simply state 3dmark != Quake III.
What you describe is running a popularity contest, not benchmarking. I would like to see the benchmarks of the many games where it "ran like total crap" for reference, though, to see how well the 3dmark problems reflected that.
Himself,
Himself said:
Strange how your criteria, which you've also stated before this, for a good benchmark is solely based on the entertainment value. Or maybe not so strange, it seems Futuremark has a similar set of priorities.
Can you see how little sense that chain of logic makes in a discussion about the validity of a benchmark? Again, this is benchmarking, not a popularity contest. 3dmark does not measure the cost of the card as part of its testing suite.
Entropy's response to Nagorak fits here so well, I'll refer you to it.
No, it is an optimization contest, not a level playing field. The problem is it is an application specific optimization contest, not a general driver optimization contest. Do we have to go over every time relative 3dmark performance of products compared to game performance has indicated this problem? Or is the jump in 3dmark performance for the GF4 that "happened" to occur around 9700 release, for example, represented in games? (An honest question, I don't own a GF4, but comments have led me to believe otherwise).
The only hope I have for 3dmark 2003 or whatever is that with the right mix of shaders and with aids to image quality comparison, optimizing for it is actually likely to benefit games in general.
Since the assertion is "invalid as an indicator of relative performance", I suggest you complete the sentence with that phrase and see if it still fits what you are trying to say. To me, when I do that it reads as something indefensible.Honestly, both ATI and Nvidia have optimizations for Quake3...GOOD, it actually runs better!
Are you deliberately missing the point, or you just don't care if new driver releases are driven by the goal of optimizing for a benchmark you sit and watch instead of games you play? You seem to enjoy the benchmark a lot, so perhaps you just don't. I hope you'll note my comments don't attempt to tackle Quake III...or atleast they don't here, I've made comments about that elsewhere before. For the purposes of this discussion, I'll simply state 3dmark != Quake III.
Doomtrooper said:It was only trounced when nothing was enabled, when FSAA and AF was enabled in some high resolution benchmarks it outperformed the Ti series by a good margin...in fact I noticed this from the very 1st reviews but as usual the 1st thing people did was talk about how low the 3Dmark scores was then proceeded to show their $400 T4600 benchmarks running with nothing on at all
remember that I didn't dig this up, but while you now are looking Parhelia scores, I would like to remind that things would be quite different, if parhelia would have stayed on original schedule (shipping before 2001 Holiday Season. P was as much as late as GF FX is now.) and/or if ATI would have decided go to 0.13µm like nVidia did. (in situation like now, they would be still the fastest card on the market and propably their decisions about staying/leaving from gamers market would have been quite different.)
But it is all about _if_
and if someone still hasn't noticed, eVGA says on their website that nVidia launches GF FX cards not untill 51 days from now: http://www.beyond3d.com/forum/viewtopic.php?t=3899
Bjorn, do you get a kick out of quoting one sentence and asking a question that makes no sense given some of the other sentences in the very same post? Like the rest of the sentences in that paragraph, and the paragraph at the end mentioning Quake III specifically?
I'm not saying don't ask the question, I'm saying ask it in a useful way. Please.
I'm not saying don't ask the question, I'm saying ask it in a useful way. Please.
demalion said:
Ok,well here's what you said after
The problem with what you say here is that the same problem arrises with other benchmark optimizations.
Will the 30% increase in Serious Sam at 1600*1200 carry over to Commanche 4 at 1024*768 ? (i think it was one of Ati's catalyst versions that had this increase in SS)
The point i was trying to make is that Quake3 and Unreal Tournament is by now just as much "a optimization contest" as 3D Mark. You might say that optimizations in Quake might lead to better performance in other Quake3 based games but i think we have seen in the past that that has not always been the case.
Bjorn said:
Do you make a habit of incomplete reading? Look at the text you quoted. Do you see perhaps why I currently feel the urge to describe you with some unflattering adjectives? Here is the other piece of text I specifically mentioned (in bold no less
):Hmm...now let's see your modified question:
Your question does not make sense given what I've stated. That's not to say there is not a version of your point that does make sense, but what I'm simply asking you to make is the ghost of an effort to actually read through my text to formulate your question in such a way that half of my point is not ignored (since I put that half of my point there for a reason...).
Let's try again. Given that I went and cut and paste and repeated text that was perfectly readable the first time, can you ask your question in such a way that doesn't ask something that I've already answered? It would help if your reply doesn't read like you stopped reading my post in the middle of a sentence and shot off a reply.
Again...Please.
- Status