The Way its Meant to be Reviewed?

oops.. that guest post above is me

 

Getting kind of OT, but I'd like to add that I haven't seen a competition's product review over at NVNews, yet I've heard of at least two that were "in works". What happened?

I have a feeling some of you think that synthetic benchmarks are for technically oriented people while game benchmarks are aimed at our average Joe.

 

Actually I do read your boards daily Max. And I do look forward to reading a review from Tim(on his own),he comes across to me as someone who could do a honest no BS review(I get this from reading his posts on nvnews)


Please don't take it personally my original post was directed at NVnews & Rage3D(and fan sites like it)as a whole,not just you as one reviewer.

 

I'd have to defer that question to MikeC. Your second statement is kind of true in a way (though not how I personally see it mind you). When I posted for ideas on my last review and the one prior you were one of the few people who requested some synthetic benchmarks, most everyone else requested more games, more IQ comparison, etc. I'm not saying anyone is right or wrong (it's all personal preference anyway)...just that's what the readers asked for.

 

NP, I don't take things personally...I mean afterall, we are just talking about games, video cards, etc...life has many more pressing matters. Granted, there is money, people's jobs on the line, etc involved...don't mean to diminish that, hope you know what I mean though.

Tim definitlely speaks his mind. I tend to bite my tongue more often than not...doesn't mean I'm biased one way or the other though. Good thoughts on your part.

 

Re: In Search of an Unbiased Review

Clay,

I believe at the heart of Mr. Sparkle's problem with NVNews.net is that it started as a NVIDIA fan site and it still uses NV in your site and domain names. You can't blame anybody for initially assuming that your site is biased towards NVIDIA products. Remember your cover mainly determines if somebody is going to continue reading the rest of the book. I myself wouldn't go to a site that has NV in it's name looking for reviews on ATI products. Same goes for Rage3D.com. In fact, I'm not sure I would go to any hardware fan site looking for reviews of any type, but that's just me. Anyway, I would suggest that if you don't want people to think your site is biased towards NVIDIA it would be a good idea if you changed your name. Or go to work for a different site like Beyond3D.com.

Tommy McClain

 

I think the discussion on whether synthetic benchmarks are needed or not is actually quite simple.

They are needed is you want to do touch the following to subject in your reviews:
1) analyze strong and weak points of a certain design
2) predict future performance.

About the first point. How would we have found out about the weak performance of the Geforce FX where it concerns PS2.0?
From the info from Nvidia? Certainly not. From the game benchmarks?

Even when synthetic benchmarsk showed weak PS2.0 performance and the first game benchmarks confirmed lots of people didn't believe that. (or did not want to admit it) Had it not been for all the synthetic benchmarks I am convinced that a lot more people would have bought Nvidia cards and would claim that the current games using PS2.0 were programmed badly. And that that isn't Nvidia's fault.

You need synthetic benchmarks to analyze certain aspects of a design. I guess that is what Dave means when he says he would like more options in 3DMark03 to work with.

I wonder how many people disagree with the first part.
A reviewer that doesn't use synthetic benchmarks at all is apparantly not interested in this.

Now the discussion is usually about the second part.

Predicting future performance with a synthetic benchmark is only possible when you know how much new features will be used in games, and what the bottlenecks will be in tomorrows games. Will it be fill-rate, or shaders?
Will people play at 60fps 1024*768 or 30fps 1600*1200 4xFSAA?

You can only do that in cooperation with the people creating the hardware and the people creating the software. And all of them have their own agenda's. And promises by hardware and software vendors won't always be kept.
Considering that, I think it is quite amazing that Futuremark did such a good job so far.

I don't think this part of the discussion will ever stop. It's based on the trust you have in the different companies. People will always have different opinions on that.

Someone mentioned that the industry is moving towars game companies partnering with IHV's and weaknesses will thus never been shown in code of those developers.
I don't really believe that that is true, but suppose that the industry is really going that way. You could even anticipate on that in a synthetic benchmark.
3DMark04 might have lots of FX12 and FP16 code in games instead of FP24/FP32, because they have seen the industry is trying to please Nvidia.
There will be of course discussion about how much they should do this. But it all comes back to what I said before. You need to hear from the software developers what they will do in future games. If they confirm this supposed trend, then your synthetic benchmark should reflect it.

 

I pretty much know the answer, though I don't get the politics

As to my suggestions for your review, I didn't specifically say synthetic benchmarks, just DX 9 applications only which we agreed there weren't many

Now that I think of performance benchmarks, it should be a mixed bag. People want to see all sorts of stuff, even things that may not be relevant at present time.

 

honestly, why is it that anytime 3DMark comes up or NVIDIA comes up in any way, NVN takes flak on the B3D forums? (The Uttar editorial thread which caused me to almost fly out to Dig's house and slap him around for a few hours with a trout comes to mind.) NVN's very honestly not biased. If I thought it were, I wouldn't be there. I'm goofy like that.

And awww, somebody actually reads my posts. I thought Stealth (and maybe very occasionally Dig) was the only one.

Anyway, back to the subject at hand. Synthetic benchmarks. Rather than spouting conspiracy theories about NVN being influenced directly by NVIDIA PR to not use them, let's take a look at why exactly we use synths.

1. We use them to gain results that are game-independent and comparable between different cards or drivers.

2. We use them to measure the performance of specific parts of a card (for example, fillrate, PS2.0 performance, and texturing).

3. We use them as a means of predicting the performance of future games.

From my conversations with Clay, I know he has no problem with using synthetic benchmarks for the first or second purpose. The third purpose, however, can be a very stupid proposition. It is, like it or not, subjective analysis. Six months from now, everything might be using UltraShadow and the 5700U will beat the crap out of the 9600XT. Would it? Hell no. But, you can't say that it won't happen.

On the other hand, though, you can predict performance just as well with games in many situations. The major exception to this is what I stated earlier in this thread, that is when a new generation of cards with new features that are not yet used in games has just been released. For this, yes, you certainly need synthetic benchmarks so you can gain some idea of a card's performance. But at this point? When it's been almost a year and a half since ATI's last major revision, and depending on whom you ask, between a year to a year and a half for NVIDIA (considering how early developer info was available for NV cards)? There are plenty of games that use PS2.0 right now; are they somehow inadequate compared to raw PS2.0 performance numbers to predict performance in PS2.0-heavy games?

The difference between using synthetic benchmarks in reviews or focusing entirely on game benchmarks is, at the crux of the matter, a semantic one. A game benchmark will take into account all aspects of a card's performance and use that in the benchmark to generate a complete view of a card. A particular benchmark may focus heavily on one particular aspect, however, so that should be noted by the reviewer.

A synthetic benchmark, on the other hand, focuses purely on one specific feature (or should. Someone who uses the game benchmarks from 3DMark in a review is rather silly, in my opinion.). Once enough synthetic benchmarks are there, it is up to the reviewer to provide the reader with a complete picture of the card's performance. A sufficiently knowledgeable reader will almost always see through any bias that a reviewer uses when interpreting synthetic benchmark scores, but most readers (even at B3D) are not. Because it relies on the reviewer for interpretation, synthetic benchmarks are far more subject to bias than game benchmarks.

How many times did you hear, "According to 3DMark, the 5900 Ultra's PS2.0 performance is weak, but the upcoming Detonator 50 drivers should fix that?" Everybody here has, repeatedly. But, how many times have you heard, "According to the Halo results, the 5900 Ultra's performance is weak even though the rest of its scores are much closer to the 9800 Pro, but the upcoming ForceWare 55 series should make Halo a lot faster?" A hypothetical and over-the-top example, sure, but the fact remains that you don't really hear the second used. If a card loses in a game benchmark, that's it, it loses. Unless there's a driver bug, which should always be noted, a reader who could not interpret synthetic scores is not going to think "Hmmm. I bet they can bring up that Tomb Raider performance!" They'll probably think, "Well, it's slower in Tomb Raider, but it's faster in UT2003." But when you have synthetic benchmarks, a reviewer who otherwise appears bias-free often shows his true colors (Kyle comes to mind. Hi Kyle. ).

So, where am I going with this? It's just a matter of on whom you are focusing for a review. B3D goes with the more technical people and uses synthetic benchmarks because of that; that's fine. A lot of the people who read NV News are gamers, so we use a lot of game benchmarks because of that; that's fine too.

Synthetic benchmarks in a vacuum (e.g., without good interpretation) mean absolutely nothing. Game benchmarks mean something, even when only numbers are presented. Synthetic benchmarks can sometimes provide more useful information than game benchmarks. It's a philosophical thing, and it's really just a personal choice. Choosing benchmarks, on the other hand, is a far more difficult situation, and I think that's something that needs to be looked at rather than the simple synthetic-versus-game discussion.

(edits: just typos and clarifying a few things. nothing major.)

 

Dang good question and I recall the same thing...any ideas Baron or MaxPower? (BTW-I read your funny posts Baron, but never the ones where you yell at me....so about half of 'em. )

 

... but didn't B3D start as Kristof's PowerVR Page or something like that?

 

I'm sorry, there was one by Pelly 100 years ago.

 

Yes indeed. Synthetics suddenly got denounced by certain websites when Nvidia told them to denounce synthetics. Of course this was because Nvidia looks particularly bad on synthetics at the moment

This has kind of been disproved. Synthetics showed GFFX to be very weak in areas like Pixel Shaders. Even with massive amounts of specialised coding, games like Half-Life 2 have shown that those same weaknesses still exist in big titles. The performance deltas are so large that they cannot be hidden or worked around, even with Nvidia's patented cheat technology.

As far as they larger question of reviews go, the problem is that many reviewers (let alone their audience) do not understand the context of benchmarks. Whether the scores come from gaming benchmarks or synthetics, most of them do not understand the relevence and meaning of the scores that they get.

Synthetics have an important role to play because they show us comparative, specialised data which is different from the information you get when running a game. They are an extreme case loading that tries to show us the maximum performance available for any given subsystem in the graphics card.

One of the reasons why Nvidia has deliberately gone out of its way to confuse the issue is because synthetics are so good at giving discrete performance numbers. With synthetics you get the discrete parts of the graphics pipeline stressed to get discrete performance numbers. That's why Nvidia have denounced them until they can find a way to get those numbers in their favour. "Framerate is king" is the only string Nvidia have to their bow, and they haven't even had that for the last year.

A gaming benchmark on the other hand gives one or two overall numbers under which Nvidia can more easily hide their cheats. With games you can argue how many frames are down to memory and CPU. You can hide poor PS2 shader performance by silently converting to PS1.1. You can drop to brilinear. You can hack the screenshot to look better than the benchmark shows when running.

Games are useful for telling you how a particular type of game runs, or compares to other hardware. Synthetics are useful for giving raw performance numbers of the various parts of the pipeline.

 

... are you not Dave? or are you HAL?

 

oh, because oh my God I was confused. but oh well, that's what I get for not being here long.

 

I actually read all the reviews and articles on this site and quote them to people who are finding vague reviews and much biased opinions in other sites. I think that untill I found out about B3D I was getting pretty confused about some differences that were being found on different hardware sites. This also led me to have to speculate about who's favoring which company and why. Where the truth lay in all the reviews I read was becoming a hard to follow chore (which almost aided me in making a very bad purchasing desicion, almost ).
I have written to some reviewers in the past about some anomalies/misinformation in their reviews and Max Power was one of the few people who took them more seriously. I think Max Power actually took my comment and updated the review, which is something other reviewers do not ussually do (even if they have proof that they are wrong/misinformed).Most reviewrs just answered in the forums with no updates to the review and the more arrogant ones just blew me off with lame remarks.
B3D is the most informative site about graphics and graphic hardware I've read so far. Although I dont always understand every bit of your reviews I take them very seriously when deciding what's true and what's "speculation" (or plain misleading) in the current graphic market.
Thank you for the great and thorough reviews and for taking the time and painstaking effort all the way through each time!

 

Without going into detail answering your post. No Games other than a couple are not really showing a weakness in Shader performance. Look at John Carmacks recent comments as just one example. While yes, FX's do have Quirky, cranky, and somewhat emmotional shader processing. If Given the Right arrangement and TLC they perform on par more or less and sometimes faster than Ati's shaders. See upcomming games like Painkiller, STALKER and others where the DEvs flat out say their shaders run faster on Nvidia hardware. Why?? for exactly the reason above. Nvidia is allowed to basically hand code or team with the developer to make the most optomized for Nvidia hardware shaders possible.

The problem is that *if* that does not happen, or the DEV simply sits down and bangs out some very standrd textbook shaders the FX will get hammered. But.. as i said... In the game enviorment of Today that is not likely to happen on the 10-20 Biggest games of the year. Which is what people will care about most.

 

Re: In Search of an Unbiased Review

Tommy,

I kind of disagree with you in the sense that all "fan sites" (whether they be for hardware, cars, sports teams, etc) have their varying share of criticisms for the product that they are fans of. Ever been to a fan site for an NBA team? I mean sometimes you'd think you were on their arch rivals fan site! For Ford Mustangs? The list goes on... You'll be sure to find criticisms as well as praises. Granted, most fan sites are likely heavy on the praise side of the fence. However, I maintain that nV News is about as close to straddling the fence as you can get. This gives nV News a lot of credibility amongst the non-jaded community out there.