Performance & IQ reviews

[Randell]

Over the last couple of years perfromance benchmarks have tended to use Normal v High Quality game settings at a range of resolutions. Recently we tend to see those plus one game benched with AA and one game benched with Aniso enabled (both usually QIIIA). If we are lucky we might get a section with AA plus aniso.

It seems to me, with all cards now fully supporting higher levels of aniso and AA that the lower resolution benchmarks should be dropped and aniso, AA and aniso plus AA scores (or at least aniso enabled) should be incorporated in the main benchmarks.

I dont know about you, but I always use AA and aniso now, and I dont intend to buy a card in the future where I dont know how its perfromance stacks up with these features on.

 

[Dave Baumann]

The issue you have there, as a reviewer, is that there are very few common settings between cards, thus making it difficulat to achieve any kinda of baseline comparison.

 

[Randell]

Well I assume a reviewer can explain the difference between MS & SS AA at least. Also different approaches are needed for a single card review compared to a graphics card comparison ound-up of performance v IQ.

In respect of aniso, the methods vary between ATI and nVidia, but ultimately arent the final outputs at say 4x & 8x similar? Once thats been explained (just the once), a link can then be provided to that explanation of the differences. Its then up to the reviewer to match up the IQ settings as best he/she feels they can (in a perfromance comparison). A review of an individual card doesnt need that extra work on matching IQ settings across cards.

Didnt Anand say in his Smoothvision driver review that the Gf3 AA needed 4x aniso to match the texture clarity of just 2x8500 supersampling?

I bet subsequent reviews havent carried this through, eg Gf3 2xAA+4x aniso v Radeon 2xAA only.

hmm then I suppose you also get into the arguement about AA quality, with ATI AA all over the place at the moment.

Well just my 2 cents on what should be attempted. This was prompted by the HardOCP Gf4Ti4200 review which clearly shows the 8500 outperforming the Ti4600 in SS:SE when aniso is enabled, yet that is only shown in one bench and not taken into account in the final round up.

 

[Doomtrooper]

Well just my 2 cents on what should be attempted. This was prompted by the HardOCP Gf4Ti4200 review which clearly shows the 8500 outperforming the Ti4600 in SS:SE when aniso is enabled, yet that is only shown in one bench and not taken into account in the final round up.

I've said this many times, and it will ALWAYS be the graph with the highest numbers that the websites BRAG about, no matter if its blurry and banding throughout the review.
It was also my concern in the LOD bias thread, of course a lower numerical default LOD card will not be as fast as a card with a higher numerical default LOD. But who cares right..its only graphics

 

[Ante P]

hehe, however their methodology sucks
they use the term 64 taps for nvidias 8x aniso and atis 16x aniso, yet in the sentence after that they mention that they're just using bilinear filtering on both cards since that's more fair

so this is 64 tap ATI aniso vs 32 tap NVIDIA aniso, or 16x vs 8x + bilinear if you scrap the stupid PR terms

nice, I was awed when I saw it
I NEVER play any games with max aniso so this is very valuable info for me
not going to get that Ti 4200 after all
I'd rather wait for prices of the 8500 to drop here in sweden (still ~350 bucks)

 

[Mintmaster]

nice, I was awed when I saw it
I NEVER play any games with max aniso so this is very valuable info for me
not going to get that Ti 4200 after all
I'd rather wait for prices of the 8500 to drop here in sweden (still ~350 bucks)

Umm.... you mean never play withOUT? or ALWAYS play with? or did you mean it the way you said it? cuz that didn't quite make sense to me.

 

[Reverend]

The answer is always the same, and I would've expected everybody to expect it and even understand it.

It all has to do with :

1) Time to come up with a review
2) If any person affiliated with a website (i.e. not necessarily the main person at the website) bothers to do such an article

As Wavey said, different "implementations" by different companies adds to the difficulty. Yes, we can still possibly do articles comparing different companies with different implementations of certain tech(s) while always using the final displayed image as the ultimate bottomline (instead of pure out-and-out "aniso performance" or "AA performance") but the problem with such an article are twofolds :

1) Will the article eventually matter/be helpful given how much later it will appear after all other websites post their "simple, first-to-market" review/article, given the amount of time it takes to do such an article? Unless you're a site that is popularly known for such articles and the majority of the public recognizes this and will wait for your article/review to come out b4 they make any purchase decision, it really is nothing other than a matter of a website's priority or a website's base "mission". Remember, numbers still is the ultimate deciding factor in the general (and majority) public's minds. For some, it is also the list of features. Educating the public about this ("what you should really look for in a review") is, so far, a losing battle. The majority mindset rules and websites cannot escape this fact. Until this mindset changes, it is very difficult for really extensive and quality reviews/articles to appear within a timeframe that may matter.

2) Related to the above - can "specialist" websites, who cares not about being "first-to-market", be around forever? Unless a website can continually be able to withstand losing money or even keep spending money on what they see as a "hobby", this is a problem when it comes to demanding folks (i.e. folks that demand a lot out of a review before they make a decision).

It is complex. If I am in a position to keep spending countless amounts of money on a website, I will keep on producing extensive "first reviews". But first I will need to know it matters and that it matters to a noticeable degree (i.e. number of folks that always refer to me/my website b4 they decide on buying). Everyone likes to be noticed, remember that.

Hopefully, in time, I will have a website without any ads and knowing that many folks will never consider any product until they read what I have to say. I don't particularly care about programmers as they will invariable tend to favor the product with the most features. That's just me, of course. As it is, I may be the "next 3dfx", tell regular folks what they really want... and then disappear.

It's just not a perfect world, is it?

 

[Mark N]

A few comments from someone who isn't all that focused on the esoteric minutia of the technology:

The people who focus most closely on the initial reviews are the ones who know the most about such things, and are unlikely to make buying decisions based solely on them and their oversimplified benchmarking and manufacturer technology brief regurgitation. I suspect most people who bought GF4 4600s as soon as they showed were ready to pull the trigger before any hard information was available, and only the most basic confirmation of specs and capability was necessary to finalize those decisions.

Most purchases of vidio cards or any other hardware don't occur within a week or two of the first reviews. If you look at the GF4s, there weren't any cards available for several weeks after the announcement and the concurrent first reviews, some of them came out much later or won't ever appear, and those reviewers were given cards long before they put up those reviews. So there is time to do a better job.

One of the problems with the current mode of reviewing based so heavily on these benchmarks is that buyers are having to rely on a bunch of statistics and not on subjective judgments of the quality of the card's performance. Who cares if the AA modes aren't identical? If you run a given game at moderate or high-quality settings or whatever is optimal and make judgments of the quality of the video presentation - image quality, smoothness, accurate reproduction of what's built into the game, etc. - isn't that what counts? These kinds of judgments are very rarely included in reviews, or at least aren't given prominence.

A perfect example of this problem was the 8500 Quake3 thing. If ATi was cheating the benchmarks by reducing the image quality, why weren't the reviewers praising the raw speed but condemning the IQ? Once known, there were a number of sites that investigated this, but even then they were very inconsistent in their conclusions and the visual evidence offered was quite questionable. So the question remains, did 10fps matter at the 190 or so the card was running, or did the IQ difference matter either? It was all right in front of these people to see, yet they seemed to have been entirely fooled until someone found evidence in the code in the drivers.

And then if you're dependent on these early reviews, you never get a very good sense of improvements made through driver updates or issues arising out of release of newer games. If a single review of a game is done at release, then they should be updated on occasion to let people know if anything has changed, at least as long as those cards are the current ones being marketed by their manufacturers. Are those first reviews of the 8500 a fair representation of what you'd get buying one now?

Aside form time/cost restraints, it seems like one of the problems is that the people doing these reviews are often not objective enough to do this job. They aren't journalists in any real sense of the word, and often don't seem too concerned with displaying very real bias. So maybe we're damned either way, and should be thankful that they stick to somewhat objective measures.

But I don't agree that numbers really make the decisions, at least comparative framerate numbers. The general public buys on much simpler criteria - general impressions obtained from a variety of sources: websites, magazines, word of mouth, even what hardware goes into the big OEM PCs. If all that is driven by numbers, then they get it second-hand. The number that does matter most for video cards is megabytes of onboard memory. And brand name is huge as well, an advantage for 3dfx at one time, now probably one for nVidia. Did benchmark numbers make the difference for nVidia back in '99 when it was TNT2 vs. Voodoo3, or was it the general impression being offered by the reviewers that the cheaper V3 was old technology missing critical features? If so, where were the meaningful comparisons of the quality of the image presentation, other than the comments about greenish smoke and fog resulting from 16-bit color rendering? And what do you remember about the G400 Max reviews, framerate or EMBM? What about the GF256 and T&L, or the V5 and no T&L, and the GF3 and the DX8 featureset?

What reviewers say about this stuff is at least as important as the benchmarks in the end, even if it's the wrong stuff to be addressing in a real-word comparison. The overall tone of the review matters, and the advantage nVidia has had is that the tone is all complimentary based on their doing all the right things with 3D featureset, and then backed up by benchmarks. But I think they miss the basic point there - how good is the card at putting on the screen the best image possible with games people will play?

 

[Randell]

well the Ti4200 isnt out yet, and people are going crazy over the numbers and betting on incredibly low prices (I've seen people in the UK reckon its going to be £120, I dont think so). There is time to do a decent comparison, make your mark with it. Otherwise all the sites look the same and TH and Anand get all the hits.

 

[Fred]

I'd like to see a reviewer who does something like the following:

Equalise framerates between the two cards (say 60fps +/- 5fps), by doing a *subjective* picking of the best (or few best) possible choices of resolution, bitdepth, antialiasing, lod, and texturefiltering (etc).

Then do a blind test.

I find it cheesy reading iamge quality reviews who forget that speed can be transformed into image quality, and then reading hundreds of people then claiming that such and such *slower* card gives a better overall image.