Killer-Kris said:
Well I have a hard time believing that we need a benchmark to tell us that our current hardware is going to perform terribly in future gaming situations.
Sorry, but the benchmark also informs people as to the fact that their hardware is equal to the task as well, so you have no point (again) to actually make. Aside from that, the use of 3dMk, like the use of any benchmark, is entirely elective in the first place. Benchmarks are generally available for people who elect to use them, and as lots of benchmarks for all sorts of things exist, then it follows that lots of people elect to use them. To that end, you are confused with respect to the word "we" as obviously you represent no one aside from yourself here.
But I suppose that is just a minor nitpick. What I believe Kyle, and most of those who oppose the use of 3DMark as a benchmark to compare cards game performance is that it is not a game of any sort. Doom3, FarCry, etc... are all games, people actually will and do play, their engines will be used in future games, their engines compute more than just pretty graphics.
So...yawn...again we have the complaint that 3dMK is a benchmark and not a game. Yet, this is true of all benchmarks, as it is equally true that games are not benchmarks--hence, your predelection to use games as benchmarks is contradictory and illogical, since if a benchmark is not a game, then obviously a game cannot be a benchmark. That is why, for instance, running Far Cry tells you nothing specifically about how your hardware will run Doom 3, or HD Tach, or 3dMK, etc., for that matter.
My consistent position has always been that there is no reason to run 3dMk in a review at the exclusion of running various games, just as there is no reason to run various games at the exclusion of 3dMk in a review. Why? Because you can run them *all* in a review, obviously, thus the imagined conflict between a game and a benchmark does not exist, since they are not mutually exclusive in the first place. There is no need to choose selectively among them, in other words, unless your intent is to selectively color your conclusions to represent a predetermined position (this practice is commonly known as "bias.")
Yes, 3DMark gives us a glimps into the future and will give us an idea of how particular cards may perform using particular algorithms. This sort of analysis IMHO has no place in a gaming orientated sight, where as IMHO it absolutely should be in a technical board review/comparison at a sight such as B3D.
Well, OK, if you are saying that [H] is not a technical site and its hardware reviews are not meant to reflect any technical realities as to the hardware they review, I suppose I might be able to agree...
It all depends on how that benchmark is being used. The information that 3DMark supplies needs to be carefully analysed and used. Yes, 3DMark03 showed us that the R300 exceled at PS2.0, and NV30 floundered. Yes, it showed us that NV30 had a very poweful stencil fillrate. What did that tell us about future games and how our hardware will run them? Well seeing as we still haven't seen a whole plethora of games that use PS2.0 very heavily, and like wise there haven't been a whole ton of games that use alot of stencil power either. Sure we do have exceptions like FarCry, D3, etc... but the vast majority of games released in the last year or two (and probably into the next year or so) still hardly make heavy enough usage of these features to show a large enough difference between the two cards. Especially so long as everyone keeps targeting the most popular graphics card in history, the GeforceMx series.
Anecdotally, I've been buying 2d and 3d graphics cards for a good bit in excess of a decade, and I've studiously avoided buying a GFMX like the plague. I have owned several other nVidia-based graphics cards, however, but never a GFMX, so as you can see what is "popular" and what is not is entirely a matter of personal opinion and perspective. For instance, I consider it a rational postulate that possibly as many as 90% of GFMX purchasers rate 3d-gaming very low on their personal totems (because, as 3d-accelerators, they certainly don't make the grade, imo.) But this is true of all "value" 3d products, not just nVidia's, of course.
In contrast to your opinion about manipulating the way a benchmark is to be used, I disagree. I prefer running it and simply reprinting the numbers it gives you, alongside running various shipping games and reprinting the numbers they give you, in tandem with IQ examinations, and then simply letting your readers make up their own minds as to what they wish to make of those numbers and IQ results. Also, I see nothing wrong with a reviewer stating his personal preferences in a conclusion that is appropriately marked as a subjective conclusion within the review, and is justified in some reasonable way. Reviews which censor certain benchmarks and/or games, and attempt to mask opinion as if it was fact, are not actually reviews, imo, but mere propaganda pieces.
And that is why you probably should not see those sort of benches in a review geared towards gamers. And sights like [H] tend to reserve those for it's CPU, Motherboard, etc... reviews. And in which case they should even then stay away from synthetic benchmarks if they are trying to show real world differences/improvements.
But you see--now you are attempting to manipulate what "gamers" see and what they don't see, and imo, that's dishonest. A "gamer's review" may consist of whatever information the reviewer wishes to include, with no exceptions, as I am aware of no laws or regulations or rules governing what may or may not be presented in a hardware review. Understand that we are talking about *hardware* reviews here and
not game reviews--which are entirely different, of course.
Well with that very same reasoning you're telling me that if I were reviewing a CPU I should run a benchmark that shows MIPs and/or MFLOPs? Those tests will give me a great deal of insite into the internal structure of the CPU, but will tell me nothing of value about most real world programs.
I'm telling you that as a reviewer is is
not your place to decide what is "valuable" and what is not when it comes to serving your readership, as the best reviews will if anything present an overflow of information and let the
reader make decisions as to what is valuable to him and what is not. IE, better too much info, by far, than not enough.
Ideally the best way to benchmark is to use the program you are interested in the performance of and run a "relatively" small relevant data set through it. And with that, we're brought back to the fact that 3DMark is not a game anyone plays.
Yes, 3dMK is a benchmark that millions of people run (again, you correctly apprise that it is not a game....
) Your suggestion would be fine if you were writing your review to meet your own specific and narrow personal needs, and writing it just to read yourself. But public reviews serve wide numbers of people who often have considerably different tastes, preferences, and desires, so that is why a hardware review should be broad as opposed to narrow, imo.
So you agree that it has no place in reviews like the ones [H] does since they focus on games? And it is anything but a "Gamer's Benchmark"? Remember games have AI, physics, sound, pathfinding, networking, etc... all factors that greatly influence performance. 3DMark is hardly a "gamer's benchmark" since it lacks most everything that makes up a game. ALL it has is graphics.
Your question makes no sense. The subject is [H]'s hardware reviews--not [H]'s game reviews. Certainly, I'm sure you know why the two categories of review are not interchangeable. What good, for instance, is a 6800U hardware review which is concerned with nothing but talking about Doom 3?....
Not much good as a *hardware* review, right? Same token, what good is a Doom 3 game review which does nothing but talk about a 6800U? Basically, a game review concerns itself with the game reviewed and is hardware agnostic, and a hardware review concerns itself with a wide variety of 3d software, including both games and benchmarks, and is game-agnostic. It's not a difficult distinction for me...
Now these bottlenecks change with time, and are probably fairly hard to predict because each team is going to have their own priorities. As far as I know, the latest 3DMark completely ignores physics, AI, sound, etc. All things most every game has, in order to even work. This greatly limits the scope of how you can use 3DMark, So once again I say that it is just about useless and quite out of place for a sight like HardOCP, who's focus is on games.
Again, 3dMk is a hardware benchmark concerned with benching the hardware used by 3d gamers, with an emphasis on direct support of the latest 3d API functionality. It's purpose is not to tell us about "games", but about the hardware we use to play those games, with an accent on the support of the latest API technology *currently* shipping in 3d products (which may not show up supported in shipping games for months after it is officially supported in the APIs and by IHVs in their 3d hardware and drivers.)
Physics, AI, and all of the rest of it are subjects for *game reviews*--not hardware reviews. Very easy distinction here...
I don't see how I've undermined anything? It has no place in a suite of benchmarks that target games, and gamers. It belongs, very much so in the suite of benchmarks that target technical aspects, algorithms, and stuff of that nature. These are two VERY different target groups with VERY different needs and uses.
Again, understand the difference between a "3d-gamer's *hardware* benchmark" and a "3d game" and you understand the difference. Trust me, it's not hard...
"Gamer's Benchmark"? It seems like they're misrepresenting themselves since it lacks everything that makes up a game other than graphics.
Let's see--3dMK exposes 3d-gaming API functionality which is supported by the 3d cards gamers are buying today, and the drivers for those cards, but is often *not supported* in shipping 3d games for months (but *will be supported* without a doubt since 3dMK mostly concerns itself with current API functionality as opposed to any future "theoretical" API functionality.) So there you have the distinction between 3dMK as a benchmark and currently shipping 3d games.
The phrase, "gamer's benchmark," means simply a "benchmark for gamers" because it is concerned with *3d-gaming API functionality* native specifically to the D3d API (the API under which 90% of all shipping 3d games ship currently.) What it does
not mean is "a game benchmark." I find it remarkable that the distinction is so troublesome for you.
Now that's something I don't care to much about nor do I want to go into.
All I'll say on that particular topic is that yes, Kyle does seem to have obsessed beyond the point of healthiness. Though the direction Kyle has been sending [H]'s reviews in, 3DMark has very little value to them. Of course by that same token [H]'s reviews have very little value to me either because they don't benchmark any games that I play (read that as they need to expand their test suite to be MUCH MUCH broader).
Well, then perhaps the solution for [H] is to drop out of doing 3d-card hardware reviews completely, then, and restrict itself to game reviews? Seems like that would fit just fine. Basically, there's just no argument [H] can bring to bear that can justify excluding 3dMk in a hardware review, because [H] isn't restricted to 3dMk but can also run all of the "real games" it wants to run at the same time. It's not an either-or proposition, and never has been.
