TWIMTBR

[Tagrineth]

I have found in the past that obsessing over framerates sometimes actually diminishes the enjoyment you get from a game. For example you play a game and turn on FPS counter in say AoM and realise it is only rendering at 20fps and you think in your head thats too low.

It's a psychological effect, and sure there are times when a game gets choppy you need extra performance but it is important to take price into consideration etc. etc. and looking at FPS all the time doesnt really tell you the overall picture of the quality of the card.

What I would do id play all my normal games and see what settings make gameplay/gfx smooth for me and write a conclusion and then benchmark it at a later time and write another conclusion.

I agree on that psychological effect thing.

Early on when I was obsessed with my V5 and its AA, I played Quake3 at 1120x864 (or something close to that), 32-bit, with 2x AA... it looked fantastic, and was playable for me. Then I turned on the FPS counter.

12fps average.

 

[ByteMe]

This topic makes me regret I never asked Sapphire for a review sample though - Fudo always insisted to me they loved GPU:RW, and I could most likely get a free 9800PRO/XT or a 9600 All-In-Wonder for "reviewing"

Uttar

NEVER accept/take/keep review samples. This makes your credibility ZERO.

 

[Tahir2]

This topic makes me regret I never asked Sapphire for a review sample though - Fudo always insisted to me they loved GPU:RW, and I could most likely get a free 9800PRO/XT or a 9600 All-In-Wonder for "reviewing"

Uttar

NEVER accept/take/keep review samples. This makes your credibility ZERO.

Then you are forced to buy all your hardware aren't you? If you buy something it is because you have already made you mind up that it is a good buy before you bought it.

You could try and accept review samples from AIB's rather than an IHV or perhaps resellers. Buying all your review items is not possible for most review sites as the people doing the reviews are often not paid anything.

 

[Reverend]

I have found in the past that obsessing over framerates sometimes actually diminishes the enjoyment you get from a game. For example you play a game and turn on FPS counter in say AoM and realise it is only rendering at 20fps and you think in your head thats too low.

It's a psychological effect, and sure there are times when a game gets choppy you need extra performance but it is important to take price into consideration etc. etc. and looking at FPS all the time doesnt really tell you the overall picture of the quality of the card.

What I would do id play all my normal games and see what settings make gameplay/gfx smooth for me and write a conclusion and then benchmark it at a later time and write another conclusion.

I agree on that psychological effect thing.

Early on when I was obsessed with my V5 and its AA, I played Quake3 at 1120x864 (or something close to that), 32-bit, with 2x AA... it looked fantastic, and was playable for me. Then I turned on the FPS counter.

12fps average.

You're a girl .... give girls some nice new lipstick and they forget that it's the face that matters, not how nice the new lipstick is. AA is like lipstick, really. Think about it.



Anyway, back to the subject matter. It's nice to see suggestions about what should be included in a review... all those resolution stuff, all those AA stuff, all those filtering stuff, all those min/avg fps stuff...

... but that's not what I asked for in my original post, is it?

 

[Tim Murray]

I write reviews for a few reasons. One, because I sometimes see other people's reviews and think to myself, "Jesus Christ, this thing sucks. I could write something twenty times better." So then I give it a go. Second, because I like it. I think that's a big part of why so many people write shoddy reviews--they like the hardware, but they don't like to review it and share their opinion. The reviews usually reflect this; they're poorly researched, make silly conclusions, and do the bare minimum of testing so they can just toss another piece of hardware in the machine.

Do I hope that a review that I write helps someone out who is buying a video card? Sure. It's not my primary reason, though. Most people are going to see maybe Anand, maybe Tom's, and maybe [H] and then base their purchase solely on those three.

Now, the limitations faced? The obvious ones. People are always going to think that you're biased, even when you're not at all, so in the back of your mind you always worry somewhat about your reputation. You can't benchmark every game at every res, and people are going to disagree with you for either putting too much emphasis on a feature or not enough emphasis on a feature. People are going to disagree with your benchmarking methods for whatever reason. You don't have enough information on how the hardware and drivers actually work sometimes to be able to actually say conclusively what the hell is going on with Benchmark X.

Basically, you can't make everyone happy. That's what it comes down to.

 

[K.I.L.E.R]

but that's not what I asked for in my original post, is it?

Did I hit the nail on the head with my post?

 

[Geo]

Anyway, back to the subject matter. It's nice to see suggestions about what should be included in a review... all those resolution stuff, all those AA stuff, all those filtering stuff, all those min/avg fps stuff...

... but that's not what I asked for in my original post, is it?

Okay, I'll play.

1. Already said, generally.

2. Follow your passion. People like me come here because this isn't the run of the mill review site. Until the HAT triumvirate blows away (heh, I like that --you can fit Hard, Anand, and Tom under a HAT) there is absolutely no need for B3D to be all things to all people. If you start trying to write to what you think people want you to write to, you do everyone a disservice. You and Dave are special. Keep on keeping on.

3. This is a total puzzler to me. I don't think Baron's answer really was relevant to the question either. Why are you asking *us* what *your* limitations are? Wouldn't you know that better than us? I hope. Or do you mean your limitations in the meaning that how much or how little of something we'll take that we aren't particularly interested in before we quit reading? If that's it, then I'd need a concrete example to look at. You have unnamed suspicions on this score? Drag 'em out into the light and let the natives chew over 'em. But note the answer to #2 above.

 

[nelg]

What are the limitations faced by folks like me and Dave when it comes to reviewing video cards?

Not enough time to surf for Porn ?

How important are these limitations?

Very ?

What can be done about these limitations?

Get an OC12 line ?

Seriously, I would ignore all games that do not tax the VPU. My blind Grandmother could do the math calculations necessary to run Quake on an abacus in real time. Do we really want to know that the 9800XT can run it at 400FPS? Also I would add price point comparisons (regardless of brand) along with the previous generation ones. I would like to see screen shots showing when one card is capable of performing an order of magnitude better than its competition. For example, card A. Runs a game at 60 FPS at 1024 x 768 while card B runs it at 58 FPS at 1280 x 1024.

 

[K.I.L.E.R]

Have any of you guys done any analysis work?
These types of questions are what I got on my business assignment.
I'm always worried that I will get them wrong or get a low mark. :?

I believe what I wrote at least answers part of the question.

 

[ByteMe]

This topic makes me regret I never asked Sapphire for a review sample though - Fudo always insisted to me they loved GPU:RW, and I could most likely get a free 9800PRO/XT or a 9600 All-In-Wonder for "reviewing"

Uttar

NEVER accept/take/keep review samples. This makes your credibility ZERO.

Then you are forced to buy all your hardware aren't you? If you buy something it is because you have already made you mind up that it is a good buy before you bought it.

You could try and accept review samples from AIB's rather than an IHV or perhaps resellers. Buying all your review items is not possible for most review sites as the people doing the reviews are often not paid anything.

Accepting any "gift" kills the credibility. This is a very black and white issue.

 

[madshi]

UT2003, Halo are obvious ones, what else?

I'd love to see some NeedForSpeed Underground benches!

 

[Miksu]

Would it be possible to do a review without any benchmark data charts thrown in?
That would be a novel concept.

Finnish gaming magazine did a review like that some years ago, when Voodoo was the only option. They didn't show any benchmarks or IQ-differences, just stated that if you want a 3D-card you can really use, buy Voodoo. It was a good review when compared to some british-magazines which were trying to give people reasons not to buy Voodoo-cards but some other (Apocalypse 3Dx? Still, it was a 3D-card from some british company). There weren't any reasons to buy anything else than Voodoo-based card but they still tried to give some reasons to the readers.

Problem then was, and to some point still is, that reviewers tried too hard to give reasons to buy a card when in reality there weren't any real reasons. "If you play game x with resolution xxx and with shadows turned to minimum, this is the card for you!" or "It's possible that with future driver-releases this will be the fastest card on earth!". I'd like to read a review which clearly says that "Card x is the best choise because it's fast in every game and will have a good support in the future." Though it's hard to make a review like that nowadays when there is so evenly matched cards from different vendors.

 

[AAlcHemY]

Hmm, maybe a stupid id, but i would like to see some power consumation numbers with a ( gpu ) review. Is always interesting to see imo.

 

[Reverend]

I don't want to sound like a tease by not clarifying exactly what it is that I wanted from you guys (the principle is a really simple one) but basically, the main thing I was wondering about is whether the type of reviews we write here at the site'o'love is informative for both sides of the "equation". It is undoubtedly nice to hear some nice comments about any one of our reviews from purchasers (or would-be purchasers) of video cards (i.e. about 95% of you guys here) and that really is the main focus of the reson why we write reviews (i.e. to better equip you with information to make a purchasing decision) but here I was sitting running through ALL the reviews we've recently been publishing and I was wondering "Well, I think we've been doing a good job for the buyers but I wonder wtf developers would think of our reviews.... surely reviews matter to them for numerous reasons...". And so... some developer feedbacks (I'll not name them yet, but will in my proposed article-cum-interview) :

I think your site has the right approach.

You're doing the right stuff.

I appreciate the effort and attention to detail that goes into creating your reviews

... feel that your site is unique in how you do things.

I have stopped visiting other sites...

All are part of longer feedbacks. I was thinking that if developers (in addition to the consumers) think we're doing a good job then surely we are looking at both sides of that proverbial fence, and looking at them correctly. I didn't ask the developers for egoistical reasons (or the risk of damaging ourselves!) -- I was genuinely concerned if what we're doing in our reviews really matters at the heart of the matter (surely you know what the matter is!), especially when certain "big" sites have avoided providing a certain kind of information that really does matter in our opinion.

 

[cthellis42]

Accepting any "gift" kills the credibility. This is a very black and white issue.

So if an IHV sense Rev a message saying "We really like the way you guys do reviews and want to support you in your endeavors. Here's a coupon to pick up any retail version of XXX card you want, so you know we're not specifically sending you a review sample, so you can judge for quality and use it for whatever reviews you want to in the future," accepting it would kill his credability instantly?

Things are NEVER black and white. Considering the vast majority of us would have no idea what goes on behind the scenes on sites like this anyway, how would we KNOW if anything like that were going on in the first place? It would seem to be that credability is to be found the way it always has--through what is reviewed and how, and what statements are made otherwise--officially or unofficially. If there are "material" issues that degrade the way products are reviewed, we will see that in the reviews themselves--not from a simple matter like this. The press is also invited and flown out to events, get in places and move about freely, or receive other handouts simply for merit of their position--does it instantaneously corrup them? Hardly.

 

[ByteMe]

Accepting any "gift" kills the credibility. This is a very black and white issue.

So if an IHV sense Rev a message saying "We really like the way you guys do reviews and want to support you in your endeavors. Here's a coupon to pick up any retail version of XXX card you want, so you know we're not specifically sending you a review sample, so you can judge for quality and use it for whatever reviews you want to in the future," accepting it would kill his credability instantly?

Things are NEVER black and white. Considering the vast majority of us would have no idea what goes on behind the scenes on sites like this anyway, how would we KNOW if anything like that were going on in the first place? It would seem to be that credability is to be found the way it always has--through what is reviewed and how, and what statements are made otherwise--officially or unofficially. If there are "material" issues that degrade the way products are reviewed, we will see that in the reviews themselves--not from a simple matter like this. The press is also invited and flown out to events, get in places and move about freely, or receive other handouts simply for merit of their position--does it instantaneously corrup them? Hardly.

I don't understand why you would make this difficult. The issue is very black and white. Just because the reality works a bit different that does NOT change the basic issue.

I do not personally know the Rev, and while he seems like a ok guy (cute butt?) if he does take "gifts" then yes he has no credibility. I don't care if these "gifts" are as small as a can of soda.

With your above example... This would work well but then the reviewer has to give the card away immediately. Maybe someone can explain this using big words so you can understand.

 

[K.I.L.E.R]

Reverend, I guess that means I answered your questions perfectly.

 

[Tahir2]

Review samples are hardly ever gifts.
They normally have to be sent back within a few days/weeks.
"Gifts" are usually given away in competitions.
I'm not talking about 'buttering the reviewer up' - I'm just saying it is impossible for a small site to personally go out and buy all the latest hardware to review. The only sites capable of this are tomshardware, anandtech and maybe someone like hardocp. Even then they are given early review samples for previews or meeting the NDA lifting deadlines.

Accepting gifts from IHV's for the intent of giving the reviewer an incentive of writing a glowing review for a good/bad product is a "black and white" issue. But I wasn't referring to that at all.

 

[Tahir2]

Rev, Beyond3D is doing its reviews correctly and there is no need to change it. The most striking difference between Beyond3D and any other site is the way you are able to UNDERSTAND the benchmark data.

Example of a usual site is:

After about a million bar graphs we get comments after each one like this:
And here we see that the Radeon is fastest with AF on but not AA.
And here we can see the box and it is a nice colour and I like it a lot.
It has a blue fan. That is really cool.
I couldnt get Doom Raider 8 to work on the Radeon and it means that the drivers are rubbish. The NVIDIA didnt run it either but that is OK because the game is rubbish anyway.
We patched the program and then we got some new drivers and we ran them instead because they are beta they are really fast.

And it goes on and on and on and even some of the bigger sites do this but perhaps not quite as bad.

Graphs etc are seen as space fillers to some sites. It gets boring very quickly.

Beyond3D takes a different approach. It runs a few benchmarks, collects some data, talks to the IHV/ISV etc and then postulates a hypothesis and sometimes proves it sometimes it doesnt. That is what I call investigative journalism and it comes across in your reviews too, very well.

 

[John Reynolds]

I don't understand why you would make this difficult. The issue is very black and white. Just because the reality works a bit different that does NOT change the basic issue.

I do not personally know the Rev, and while he seems like a ok guy (cute butt?) if he does take "gifts" then yes he has no credibility. I don't care if these "gifts" are as small as a can of soda.

With your above example... This would work well but then the reviewer has to give the card away immediately. Maybe someone can explain this using big words so you can understand.

Then pretty much every review you've ever read is worthless. What? You think Ebert pays to get into every movie he reviews? You think the Sun-Times goes out and buys a copy of the latest King horror novel for review?

That said, there have been studies done on the psychological impact 'free' review samples have on the reviewer, which is why companies are so willing to send them. But it is pretty much standard practice across all entertainment markets.