TWIMTBR

R

Reverend

Banned
... = The Way It's Meant To Be Reviewed.

So... if you were a reviewer, how would you review a video card? What software would you use? Why? Why wouldn't you use certain software?

What would be the primary reason you'd write a review? For the benefit of the consumers? For your very own interests insofar as your interest in video cards are concerned?

What are the limitations faced by folks like me and Dave when it comes to reviewing video cards? How important are these limitations? What can be done about these limitations?

This post, and the questions I posed, should serve as an appetizer (and probably an illuminating one, depending on the replies here) for a forthcoming mini-article.
 
Do NOT use any popular software or benchmarks. We could read/see them at many other places. Make all custom "demos" for benching. Use fraps and show the entire time/frame chart.

As an example one of the older software programs to use TOTAL ANNIHILATION. But don't use all old programs.

I'd also like to see linux benchmarks.
 
Myself, I would use as much software as possible without going insane. ;) Popular titles SHOULD be used because people are concerned with their performance in said titles, but I also think that one needs a good mix of types of games, as well as popularity of them. I would probably include a few "wild card" games that come out of left field and are never predictable, because it tests the IHV's support of left-field titles. (Avoiding craptacular examples, of course.) I would NOT avoid synthetic benchmarks, as they certainly have their place as well. I would pull a wide range of them as well.

The most important things, imho, are the commentary. Let people know what certain games and certain cards stress, and put things in context of other results and the overall business environment.

I would likely want an in-depth, constantly updated image quality test that runs parallel, and is invoked before talking about performance, because there is no way to judge performance at quality properly without knowing how to COMPARE said quality.

The important part is to make sure the explanations are there. What software is being used and why, what it stresses, what it shows off on a card, how to measure performance gains to itself and its competition... What hardware is being used, whether it is retail or sent directly as an eval card (and likely higher quality)... Basically, ever pertinant piece of information.

In the end, you can't stop people from just looking at charts or skipping to the conclusion, but I don't think reviewers should be pandering to that crowd. These are highly technical devices--and that means the more information that's out there, the better.
 
... = The Way It's Meant To Be Reviewed.

So... if you were a reviewer, how would you review a video card? What software would you use? Why? Why wouldn't you use certain software?


I would review my video card extensively on every game I play. I would do so because I'm only going to be using my video to play games I would normally play, there's no point in using "certain software" because I'm not interested in the performance/IQ of "certain software". I'm assuming "certain software" == 3DMark?

I would test my games using max IQ possible because that's how I play my games.



What would be the primary reason you'd write a review? For the benefit of the consumers? For your very own interests insofar as your interest in video cards are concerned?


Primary reason I would write a review would be so I can share my results with others, they may be helpful to others.


What are the limitations faced by folks like me and Dave when it comes to reviewing video cards? How important are these limitations? What can be done about these limitations?


You guys do in depth reviews which I would consider very well done and professional. You guys always test the latest and greatest, I believe more reviews should be done using lower end hardware like CPUs and video cards. There are not many people who can afford much of the stuff Beyond3D reviews so your reviews are limited to the enthusiasts and not the average gamer, so maybe you guys can test games using a variety of CPUs and videocards? Don't forget the videocards are mainly influenced by the CPU. Your reviews don't accurately measure the performance of the video card under the most common system configurations.


This post, and the questions I posed, should serve as an appetizer (and probably an illuminating one, depending on the replies here) for a forthcoming mini-article.


An article about what exactly?
 
- 4x AA and 8x AF only (when testing top level cards at least)
- only 1024x768 and 1600x1200
- as many brand new games as possible (e.g. have not seen any Nfs Underground benches yet)
- as many beta/preview demos of future games as possible
- only custom made timedemos (if possible)
- only using WHQL drivers

Then I'd retest all results with the AntiDetect script. For all discrepencies that AntiDetect reveals I'd ask for comment from the IHVs (probably I wouldn't get any informative replies about that, but I'd ask nevertheless).

At the end of each review I would bitterly complain about:
(1) The forced brilinear filtering (-> NVidia).
(2) The missing CP option to force trilinear filtering on all texture stages when doing AF (-> NVidia + ATI).
 
Lol, Simon/Dio/Andypski. Talk about getting picky about useless stuff, lol ;) Nice way to say you're following this topic a bit though :p ( not like you didn't follow just about all topics here :) )

This topic makes me regret I never asked Sapphire for a review sample though - Fudo always insisted to me they loved GPU:RW, and I could most likely get a free 9800PRO/XT or a 9600 All-In-Wonder for "reviewing" :LOL:

Anyway, regaring this...
I love this way B3D does it, mixing synthetic and games benchmarks. But I'd take a different approach when it comes to mixing the two, AND I'd compare competing cards more ( all IHVs, and maybe a card which a slightly lower/higher pricetag of the same IHV to see if it's worth the extra cash ). I'd prefer a rather... untraditional approach, too:

1) Run synthetic benchmarks and explain what they stress, how many current games stress that and how likely future ones are to stress it.
2) Give the names of the game benchmarks, then say what they stress. Based on the synthetic benchmarks, make speculations as to which card will be ahead in them. This should be done BEFORE benchmarking the games, or it's no fun! :p
3) Run games benchmarks, if the expected results are not met, try to explain them.
4) Look at future, mostly highly-anticipated games and based on the scores, try to determine which cards are most likely to be ahead.

Yes, yes, I know it's not an easy thing to do. And no, most review sites couldn't do it, simply because their reviewers aren't competent enough to do it. But I do believe Rev or Dave would be capable of such a review; hopefully my respect for them isn't underserved though! :devilish:


Uttar
 
The Baron said:
What a nice topic to have as I'm about to begin writing a review :)
Click to expand...
I was thinking the same thing, but I'm finishing one up. :)

Think charting fraps min/max/avg for games on the same card on a couple of different rigs on a selection of games as well as the standard synthetics will give a nice rounded review? (I'm not doing the Brent scatter-graphs of fps. I liked the idea but it seemed awful confusing, I like the simplicity of the min/max/avg. It gets the info across I feel you need without making it over-complex. :) )
 
It seems to me that if recent events have proved anything, it is that you need both popular and lesser-known titles. If a card is competitive or dominant in the former and gets its ass handed to it in the latter, that is a valuable piece of information.

And the lesser knowns need to revolve a bit to make it harder for the driver teams to zero in you. Maybe you have a stable of twenty or so and draw lots for which five to use in any given review?
 
I would be certain to include the following resolutions: 1024x768, 1280x1024, 1600x1200. The 1280x1024 or 1280x960 is a must, as more people are running on LCDs with that as a native resolution. I'd then continue in B3D fashion and show the impact of adding IQ (AA with AF) to the mix.

If possible, run the video card in several HDTV resolutions (ala 480p/720p/1080i). Along with that, try to measure the noise-level of the card. The last two points I have rarely seen but will provide a unique perspective for the HTPC-going crowd.

Perhaps a review of the drivers (one per manufacturer per major functionality change) should be done as well. It would be nice to have a reference perspective on what features are included with Nvidia's or ATI's set (including their nvDvd or MultiMedia Center, nView or HydraVision, etc...).

As for what titles, among seeing the 'typical' FPS games, I'd love to see some racing and flight sims used. As is standard, custome time-demos or fraps logs of gameplay will need to suffice. I rarely care about max-fps as min and typical fps are better indication of smoothness and playability. Also, even though its very subjective, I'd like the reviewers opinion on what settings were playable and which ones weren't. There's been a few games that indicate low fps (< 30) but feel completely smooth and playable.
 
First, I'd love to get rid of the 4xAA/8xAF standard. That really annoys me because it's both artificial and biased towards the Gf4. A Radeon 9800XT user is going to want to run 6xAA/16xAF if they can, not 4x/8x, so they need to be shown that.

What I'd like to see are visual quality screens showing No AA/AF, median levels, and then Max AA/AF. That way we can see the difference. Then show min/max/avg framerates for each one. Remember, the important things about AA and AF are how it looks and the performance hit it costs. For most tests I'd stick to 1024x768 and 1280x960/1024 because those are the most common gaming resolutions. You could go up to 1600x1200 and down to 800x600 but those seem to be less common with additional eye candy.

A good mix of OpenGL and D3D apps is also a must, but others have covered that
 
Would it be possible to do a review without any benchmark data charts thrown in?
That would be a novel concept.
 
One thing I would DEFINITELY like to see tested in this environment are comparing those games what have industry-made benchmark procedures, and comparing them to FRAPS runthroughs of similar nature and representing other parts of a game to gain insight on just how those types of benches line up to in-game performance as well. Obviously harder to do on games which allow no personal demo-recording, but even without it can be averaged and still come up with some useful info.

What with people's "game benchmark uber alles" attitude since the 3DMark fiasco broke and has perpetuated, it would be a good idea to actually TAKE a solid look at them.
 
Tahir said:
Would it be possible to do a review without any benchmark data charts thrown in?
That would be a novel concept.
Click to expand...
What, so no benchmarks per se at all? Interesting thought. Just compare IQ and such, reviewer's subjective opinion of the card. I think with the 5700 Ultra review I'm working on, I'll arrange it like so:

Physical nature of the card, layout, etc.
IQ comparisons between 5700 Ultra and 9600 Pro--games TBD (UT2003, Halo are obvious ones, what else?)
synthetic benchmarks to test specific features
subjective analysis of the card--how it performs, what I like, what I don't like, etc.
game benchmarks
conclusion

what do people think about that? and ideas for benchmarks would be a big plus right now--I'm sick of the standard UT2003/Splinter Cell/some sort of OGL/etc. combination, but I don't really know what else to try...
 
I have found in the past that obsessing over framerates sometimes actually diminishes the enjoyment you get from a game. For example you play a game and turn on FPS counter in say AoM and realise it is only rendering at 20fps and you think in your head thats too low.

It's a psychological effect, and sure there are times when a game gets choppy you need extra performance but it is important to take price into consideration etc. etc. and looking at FPS all the time doesnt really tell you the overall picture of the quality of the card.

What I would do id play all my normal games and see what settings make gameplay/gfx smooth for me and write a conclusion and then benchmark it at a later time and write another conclusion.
 
A good idea would be comparative screenshots of smooth gameplay on the two cards. Run the highest settings that give you good performance and let us know not only what those settings are, but what it looks like.
 
You must log in or register to reply here.

Similar threads

S
Discussion about lighting (usage of lighting techniques).
2
Replies
21
Views
1K
Frenetic Pony
F
Shifty Geezer
Feedback wanted on game project [PC Web] *self promotion
Replies
19
Views
2K
Shifty Geezer
Shifty Geezer
Cyan
Any fan of Tower Defense games here? I mean games like Defense Grid or the wonderful unrepeatable Bloons TD6.
2
Replies
30
Views
5K
Silent_Buddha
S
I
"Material/shader-space lighting" and softness in modern rendering
Replies
9
Views
1K
JoeJ
J
J
Xbox Thought Experiment
2
Replies
27
Views
2K
vjPiedPiper
V
Back
Top