The Way its Meant to be Reviewed?

Good gods, I see why that person didn't log in to make those posts!

 

Thank you for what you have done for me. I grabbed this link via Anandtech's artical and read the comments posted. You have sealed the nail in the coffin for me as I was correct with my first observation, that B3D is a spin zone fuled by the optimistic underdog spirit. The more I read here, the futher from reality I feel. Keep believing that synthetic benchmarking like the kind Futuremark builds have a use. My optimistic spirit suggest Futuremark will be around for another year before they cash out. Then again, I am just being optimistic.

 

BZB is falling into the trap of assuming synthetic benchmark performance equals real-world performance, which isn't necessarily true at all for various reasons set forth earlier in the thread. And our good friend the Guest is falling into the trap of believing that synthetic benchmarks have no relation to real-world performance. They're both wrong. Oh well, time to go.

 

When a synthetic test can provide information that a game cannot it is beneficial. When a synthetic test can more clearly define a VPU’s strengths and weaknesses it should be used. It is up to reviewers to ignore this information at their own peril. We all know [H] shares the stance of “only real games tell you how a card performs gamesâ€￾ but look at how this distorted their conclusion of the BFG5600u......

This was done even though they had these results.

 

mmmph...I just adore blanket statements. I don't mean to take direct exception to this and no offense but just for the record, no one at nV News or at NVIDIA has ever called me "Polly" and/or tossed me a cracker. We should get back on track as others have eloquently stated in that this is not about "synthetics vs. games" benchmarking. It's about deciding on what synthetics (and games) are best to use in reviews. The mix is good, the mix is necessary, balance is good. In hindsight my first review was half-baked because I didn't include any synthetics for example. (At the time though everything was going down though that made 3DM03 results very suspect.)

To your last sentence. Nah, they're really not so weak as you put it. Perhaps on HL2 or some other future game but that remains to be seen. For now, however, I can play any game I own on either my ATI 9800Pro or NV 5900U and any differences are negligable. They both have their pros and cons...it's a wash IMO at the end of the day.

 

Not at all. Synthetics only compliment games, because they do different jobs from games. I only addressed synthetics because that's what was being rubbished. You have to understand that synthetics are a way of stressing discrete parts of the graphics card to the maximum. This is quite different from games, and so give different metrics about relative performance.

The trick is to be able to tell what these metrics will mean when these same parts of the graphics card are used for the quite different "real world" of playing games. This information is at least as, or more useful than looking at the performance of the latest Quake-engined games and then trying to apply it to UT2K3, HALO, NFS:U, or any other different games out in the "real world". Even more so for upcoming games you will want to be playing on your newly purchased card in the next 12 months.

 

Since I prolly helped to foster this 3DMark03 talk I'd like to suggest we quick kicking the dead horse (not that FM's product is a dead horse, you know what I mean). Dave keeps stressing that synthetics aren't soley 3DMark03 and I agree so let's focus on what synthetics should be used...when and why along with games in reviews. Shall we? Seems much more productive.

BTW Dave, you did have "3DMark03" as the fifth word of your first sentence in the initial post here so that's why I focused on it to begin with.

 

Hehe, it s very fun to read the latest Nv recruit coming to try to show the big boy he is

 

Who is the newest Nv recruit? :?

 

I think he's talking about our anonymous 'Guest' friend who appears to be, somewhat perversely, completely misunderstanding what the article and this discussion is about.

 

OIC

 

I think BB was just shy about registering and using his real name.

 

Yeah, I am saying they're fundamentally wrong. We're really talking about a 'chicken or the egg' question here. I think it is a little more resolvable than that though. There has been a concerted effort by some to discredit synthetic benchmarks in general and instead convince the review industry to focus solely (or nearly exclusively) on 'real games.' So your readers have been telling you they want more actual games? Well of course they have. They've been told over and over again that they are the only legitimate way of measuring performance. There have been few voices of dissent at actual sites (as opposed to forums) and they've been far between. That's why I think this article is so important. The job of the reviewer is to inform the reader about the subject of the review. To do that, one has to cut through marketing spin. Synthetic benchmarks that stress specific and limited parts of graphics architectures can do that. Games aren't so good in this regard.

Certainly you have to be responsive to the wishes of your readers. And certainly you should use lots of games. Should you run lots of game tests? Heck yeah. That's important for current games and for people who play various genres, etc. So you're saying that giving the readers what they want (games) isn't prescriptive because they've requested it. I say it is prescriptive because their requests are based on misconceptions that have been propogated (intentionally by some, unintentionally by others) for a while now. Those misconceptions are that synthetic benchmarks are inherently unreliable and do not reflect actual games.

Okay, the unique (i.e. original) damage has already been done in that this view has been loudly advocated with little dissent and is now widely viewed as legitimate. Solely listening to your readers (and I know you aren't only doing this) and piling on the real game benchmarks reinforces this dominant paradigm. To the degree that people look at reviews not only as telling them what hardware is good, but also as telling them what is a good way of determining if hardware is good (and I'd say reviews have this effect of everyone if they realize it or not), then it is prescriptive on some level.

The argument that 'normal gamers don't need to care about synthetics; that stuff is for the really technical enthusiasts who want to get into the nitty gritty' is distinct but related. That view doesn't indict the validity of synthetics (rather it implies they are quite useful, but only to experts in the field), but it does discourage their use by reviews and says that normal people shouldn't care about their results. (And most people are disinclined to think of themselves as abnormal...) Reviewers are opinion leaders. So when they say (actually say it or communicate it through their actions, i.e. by not using synthetics or by using them by not evaluating them when drawing conclusions, etc.) that synthetics don't matter, that leads readers to think synthetics don't matter. I would argue that such a view makes for inadvisable decision-making. The idea that only the ubernerds need synthetics goes a step further. It not only presents an incomplete picture, but it also tells people to actively disregard the rest of the picture should they accidently run across it. At least, that'd be my argument.

Yeah, I should have made that more clear. By "same technologies" I'm not talking about identical game engines or the exact same shaders. I'm talking about methods of doing things, not the actual things that get done. I'm saying shadermark is valid and useful because games will use shaders that are similar in form or function in the future. For example, to the degree that virtually every test of DX9 PS2.0 shaders has shown NV3x to have some trouble off the bat, I would say that is an accurate representation of how the average game that uses PS2.0 will run on NV3x.

And Rev, I think we did read the article. The parts you quoted just seemed so self evident that I didn't feel a need to comment on them.

 

Throwing type I and type II errors around, huh? If you're a developer doing preliminary research into what features you're going to put into your next engine, what will give you a better general predictor of perfomance: a poorly developed alpha that uses the exact features you're looking at but has low validity or a fully realized synthetic that tests factors similar to the ones your interested in, but not the ones you want, that has high validity?

You're right that driver revisions and developer tricks will skew these results but if something doesn't perform somewhat reasonably before these factors come into effect, no one or very few people are going to use them.

 

Nice post hmmm, some stuff to chew on (in a good sense) there for a while.

 

Ah, one thing I forgot to respond to. Not sure exactly what you're trying to imply there but the fact is I really do just listen to my readers when it comes to what will be in my review. I'm nobody's pawn in that regard. It's funny how some people (not saying you exactly here) subscribe to the conspiracy theory across the board. Sure, cruddy stuff does go on. However, I'll stamp my foot and spit quite a bit before I let anyone suggest that I (or nV News as a whole) falls into that category. No one from NVIDIA has ever even contacted me directly regarding what should/should not go into my reviews. No IHV has either, sure they make suggestions about key selling points, differences from competition, etc but that's it. And yep, my reviews are edited...but they're never altered...if they were I'd be gone in two seconds. Errr...I know talk is cheap but it just gets my goat to be associated with other sites/practices/etc when those associations are, in fact, false. Again, not pointing and shaking my fist directly at you hmmm...just venting a bit. (sorry for more nV News talk Dave...I'll ride off into the sunset soon).

 

I coming late to the party , but I have one importan thing to point out.

Never ask readers how to benchmark cards as they mostly dont understand how it works. It is reviewers job to find out the best way to benchmarking.

Readers (with exceptions) usually know nothing except some marketing slogans that they heard. That is why I'm constantly bugged by readers if FX5200 with 256 MB memory and 64-bit bus is good buy as it is cheap and has twice the memory of Radeon 9600/9600 Pro that I suggested as best buy.

Further more many readers, at least ones participating in forums are biased heavily and most of them want to prove that their card is the best.

I know only a couple of persons that where saying synthetics sucks prior to 3DMark/nVidia saga and they were saying it because they heard it without thinking about situations and what syntetics can bring. After nVidias anti 3DMark03 propaganda it changed rapidly and I have to explan to some of my collegues that also review hardware that it is not 3DMarks fault that only nVidias score drops and so on.........

So what I want to say is that people generaly dont know a lot about graphics cards and are very easily influenced so it is up to reviewers to judge.

I personaly love syntetics as game benchmarks tell me only how this game will run that game and it is very difficoult to make some sensible conclusions if card for instance obtains 170 fps in UT2K3 flyby. So I run every synthetic test that I have plus only a couple of games that are either widely used or are interesting for some reason.

Zvekan

P.S. Sorry about my english

 

Troll Alert!
Bweeeep! Bweeeep! Bweeep! Bweeeep! Bweeep! Bweeeeep! Bweeeeep!

Please ignore the above authors postings.
This individual may be a [h]ard worker, but is [h]ard[h]eaded to the extent that they are willing to ignore all information contrary to their pre-existing point of (n)View.
Especially if the New information contradicts their cozy relatioNships with IHV PR people.

 

Wow, that's pretty good spelling Brian. Not perfect but quite the improvement compared to emails.

Look, you don't think synthetic benchmarks have any value at all. You seem to think the only synthetic benchmark around are those made by Futuremark. Thank you for your opinion. Now, please go back to doing what you're paid to do.

Wait a minute... all that you've posted in this thread is what you're paid to do, isn't it?