WaltC said:
Killer-Kris said:
...
Please don't make an idiotic statement saying "Far Cry sucks because it doesn't specifically tell me how my box will run Doom 3," because unlike 3DMark, FarCry will tell you how well FarCry runs on that particular hardware and Doom3 will tell you how well Doom3 runs. What does 3DMark tell you about games?
3dMk is what? Is it a benchmark designed to tell you how every game will run on your system, or is it a "gamer's benchmark" designed to provide you with information as to how your current
hardware is
likely to perform in software gaming situations
which use the latest in API feature support, support that might be months or even a year or more away from actually being implemented in shipping 3d games?
Well I have a hard time believing that we need a benchmark to tell us that our current hardware is going to perform terribly in future gaming situations. But I suppose that is just a minor nitpick. What I believe Kyle, and most of those who oppose the use of 3DMark as a benchmark to compare cards game performance is that it is not a game of any sort. Doom3, FarCry, etc... are all games, people actually will and do play, their engines will be used in future games, their engines compute more than just pretty graphics.
Yes, 3DMark gives us a glimps into the future and will give us an idea of how particular cards may perform using particular algorithms. This sort of analysis IMHO has no place in a gaming orientated sight, where as IMHO it absolutely should be in a technical board review/comparison at a sight such as B3D.
Of course, the correct answer is the latter, imo. The API functionality 3dMk supports is both real and current--the implementation of that API feature support in shipping games is likely to be much later in coming, however. Does the fact that a benchmark supports current API functionality that won't be seen in shipping games for months, but will indeed be seen in them eventually because its already in the API, somehow invalidate 3dMK as a benchmark?
It all depends on how that benchmark is being used. The information that 3DMark supplies needs to be carefully analysed and used. Yes, 3DMark03 showed us that the R300 exceled at PS2.0, and NV30 floundered. Yes, it showed us that NV30 had a very poweful stencil fillrate. What did that tell us about future games and how our hardware will run them? Well seeing as we still haven't seen a whole plethora of games that use PS2.0 very heavily, and like wise there haven't been a whole ton of games that use alot of stencil power either. Sure we do have exceptions like FarCry, D3, etc... but the vast majority of games released in the last year or two (and probably into the next year or so) still hardly make heavy enough usage of these features to show a large enough difference between the two cards. Especially so long as everyone keeps targeting the most popular graphics card in history, the GeforceMx series.
Understand that
I didn't say, "3dMK sucks because it tells me nothing specifically about how Doom 3 runs on my box," but [H] said it, and I was merely illustrating what a ridiculous, inappropriate statement that is. It's exactly as ridiculous as saying, "Far Cry sucks because it tell me nothing specific as to how my box will run Doom 3."
While Kyle didn't state it in necesarily the best manor, what he was likely meaning was that no one plays 3DMark, it's not a game and it should not play a very large role in determining whether a piece of hardware plays a game well or not.
No benchmark of any type, be it a cpu bench, a hard drive bench, a hard-drive controller bench, a ram bus bench, ad infinitum, wil ever, under any circumstances, tell you how your box will run
any specific 3d game.
And that is why you probably should not see those sort of benches in a review geared towards gamers. And sights like [H] tend to reserve those for it's CPU, Motherboard, etc... reviews. And in which case they should even then stay away from synthetic benchmarks if they are trying to show real world differences/improvements.
Therefore, if that is a valid reason for tossing out 3dMk, then all other benchmarks are also meaningless and worthless for the same reason.
Well with that very same reasoning you're telling me that if I were reviewing a CPU I should run a benchmark that shows MIPs and/or MFLOPs? Those tests will give me a great deal of insite into the internal structure of the CPU, but will tell me nothing of value about most real world programs.
Ideally the best way to benchmark is to use the program you are interested in the performance of and run a "relatively" small relevant data set through it. And with that, we're brought back to the fact that 3DMark is not a game anyone plays.
But it's easy as pi to illustrate what's wrong with that notion, because none of these benchmarks, including 3dMK, is designed for the express purpose of informing the user as to how any specific game will run on his system. Instead, these benchmarks, including 3dMk, are designed to provide other information about the tested hardware which is both general and generic in nature. That, of course, is the difference between a "benchmark" and a "game."
So you agree that it has no place in reviews like the ones [H] does since they focus on games? And it is anything but a "Gamer's Benchmark"? Remember games have AI, physics, sound, pathfinding, networking, etc... all factors that greatly influence performance. 3DMark is hardly a "gamer's benchmark" since it lacks most everything that makes up a game. ALL it has is graphics.
Now these bottlenecks change with time, and are probably fairly hard to predict because each team is going to have their own priorities. As far as I know, the latest 3DMark completely ignores physics, AI, sound, etc. All things most every game has, in order to even work. This greatly limits the scope of how you can use 3DMark, So once again I say that it is just about useless and quite out of place for a sight like HardOCP, who's focus is on games.
Now with that said, that's not to mean that 3DMark isn't with out it's uses. I for one believe that 3DMark is and will forever be infinitely invaluable to sites like B3D because it gives you a peak into a possible future, and shows hardware strengths/weaknesses that we would otherwise never see. It allows you to in a controlled environment to change settings variable by variable in order to isolate particular behavior and is just an all around excellent tool for that sort of work.
Good observation, which of course undermines everything else you said previously, but duplicity of thought seems to be an abiding characteristic surrounding this topic...
Yes, 3dMK is a benchmark, not a game, and should be used and thought of only in that capacity.
I don't see how I've undermined anything? It has no place in a suite of benchmarks that target games, and gamers. It belongs, very much so in the suite of benchmarks that target technical aspects, algorithms, and stuff of that nature. These are two VERY different target groups with VERY different needs and uses.
I'm not aware of FutureMark ever representing it differently.
"Gamer's Benchmark"? It seems like they're misrepresenting themselves since it lacks everything that makes up a game other than graphics.
Let's call a spade a spade, shall we? Prior to nVidia's "War on 3dMk"...
Now that's something I don't care to much about nor do I want to go into.
All I'll say on that particular topic is that yes, Kyle does seem to have obsessed beyond the point of healthiness. Though the direction Kyle has been sending [H]'s reviews in, 3DMark has very little value to them. Of course by that same token [H]'s reviews have very little value to me either because they don't benchmark any games that I play (read that as they need to expand their test suite to be MUCH MUCH broader).