First few GFX benches

Status
Not open for further replies.
Reverend said:
To those that have always looked at PC graphics as nothing more than "something that displays images for me to play games", both aliasing and blurry filtering is perhaps not noticed, or maybe sub-consciously they do notice them ("Hmm... that's not like in real life") but they have "learned" to accept them as part of PC graphics or have learned to ignore them (both accepting and ignoring having the same result). i.e. "That's the way it is". These folks have much "unlearning" to do since they do not know these really are imperfections and are not to be accepted and/or ignored.

I'd agree with you Rev.
I have a copuple of friends that used to play without vsync on. One time at a LAN, i commented on the terrible tearing i noticed. They were like, WTF are you talking about? I explained, and demostrated the superiority of vsync :)
Two days later, i got an email from one of them cursing me - he now noticed the tearing in all his games :)
 
There can be a problem if people think that numbers are more important than IQ.
Because if the companies think that they need high numbers(fps) to beat competitors they will be tempted to cheat.
They could create implementations of AA and anisotropic that have low IQ because they want to get higher framerates.
A competing company with much better IQ will suffer if hardware sites only compare numbers.

Because of that I think it´s very important to compare IQ. If card 1 looks much worse compared to card 2 when using 4*AA it is very relevant to know and talk about if you want to explain why card 1 is faster.
Because in that case card1 has sacrificed the IQ to get higher framerates.

Regards!
 
I like objective comparisons.

Subjective comparisons always end up in "but you're wrong". I'm sure the 4x vs. 4x or 6x "debates" will be no end to the joy in this world when benchmarks start flying. "This card beats that card" "no it doesn't, you need to compare X vs. Y, instead" "Yeah, whatever, my X is the same is your X, or just slighly worse" "no way! Your Y is barely my X. You should REALLY be using Z instead" And on and on and on.

Of course, not that my little rant will change anything--because the output isn't the same, you need to get them the same before you can compare the two. The only problem is you'll NEVER find agreement on what the same actually is.
 
The problem,IMO, is that if you compare two things that are very different it is in a way no longer objective.
Because if what company A calls featureX in it´s drivers is very different compared to featureX in company B´s drivers the difference in framerate is no longer a fair way to compare.

The best way to compare, I think, is to explain what featureX in company A`s driver really is. How it´s done. How it works. The differece compared to B.
Then screen shots might help the reader to understand better.

Regards!
 
On the topic of 4XAA comparisons of NV30 vs R300:

Okay, so I get it that the concensus here is that such comparison would be unfair to ATI IQ-wise BUT... Given the much larger bandwidth requirements of 4XAA over 2XAA, I would also expect the R300, with all of its bandwidth, to win the FPS race as well vis-a-vis the NV30, with its more limited bandwidth. Or am I incorrect in this expectation?

On the topic of Reviews:

Reverend, has your R300 review ever been published at ve3d.com? Or anywhere else, for that matter? If not then I, for one, would still be interested to see it.

Hrmmm... that reminds me, whatever happened to Ben6's review of R300? Was that ever published? If so, then I missed it and would appreciate a link.

EDIT: Found it. Thx Doomtrooper.

And does anyone else miss seeing competitors' benchmarks in B3D reviews? It seems that all of the latest B3D reviews focus only on products from the same IHV. IE: R9700 vs R9000 vs R8500 or GF4 vs GF3 vs GF4MX but not R9700 vs GF4. While the B3D reviews are very informative, I do miss seeing 'shootouts' between all of the players.
 
My only beef with reviewers, is they are forced into time constraints.

In an ideal world, benchmarking can be made much more scientific. Ergo a group of scientists could probably achieve the needed accuracy to say such and such a setting on card X indisputably has a lead over card Y.

Aliasing and AF are fundamentally signal processing quantities that are in principle objective and quantifiable. Where the problem becomes too complex to be dealt analytically, large sample blind tests should be available to make the difference.

So I only buy the *everything is subjective* line of argumentation so far as lack of time and effort makes it so.
 
RussSchultz said:
I like objective comparisons.

Subjective comparisons always end up in "but you're wrong". I'm sure the 4x vs. 4x or 6x "debates" will be no end to the joy in this world when benchmarks start flying. "This card beats that card" "no it doesn't, you need to compare X vs. Y, instead" "Yeah, whatever, my X is the same is your X, or just slighly worse" "no way! Your Y is barely my X. You should REALLY be using Z instead" And on and on and on.

Of course, not that my little rant will change anything--because the output isn't the same, you need to get them the same before you can compare the two. The only problem is you'll NEVER find agreement on what the same actually is.
Then the best solution is to set the best IQ for a given framerate target.
Say, UT2k3, framerate of 60fps avg.
Now, give the best IQ of each card that gives approx this average.
whichever looks best wins :)
 
790 said:
2X AA is a huge difference. Simply upping the resolution does not negate the need for AA, and never will.
I disagree. Upping the resolution is a far better alternative for me personally with current games. Everyone else's mileage may vary.
 
Althornin said:
RussSchultz said:
I like objective comparisons.

Subjective comparisons always end up in "but you're wrong". I'm sure the 4x vs. 4x or 6x "debates" will be no end to the joy in this world when benchmarks start flying. "This card beats that card" "no it doesn't, you need to compare X vs. Y, instead" "Yeah, whatever, my X is the same is your X, or just slighly worse" "no way! Your Y is barely my X. You should REALLY be using Z instead" And on and on and on.

Of course, not that my little rant will change anything--because the output isn't the same, you need to get them the same before you can compare the two. The only problem is you'll NEVER find agreement on what the same actually is.
Then the best solution is to set the best IQ for a given framerate target.
Say, UT2k3, framerate of 60fps avg.
Now, give the best IQ of each card that gives approx this average.
whichever looks best wins :)

Now there's an idea.
 
Slides said:
790 said:
2X AA is a huge difference. Simply upping the resolution does not negate the need for AA, and never will.
I disagree. Upping the resolution is a far better alternative for me personally with current games. Everyone else's mileage may vary.

That was my solution until getting an LCD monitor at 1024x768.

(Plus, I think in general, 1024 or 1280 is the highest real resolution most monitors can put out. excepting of course the high end 19 and 20" CRTs/LCDs)
 
A lot of 3d Reviews today lack information and this truly boils down to lack of testing, originality, and 3d passion. IQ is a very important aspect of immersion and if this important aspect is not presented fairly, well, a reader will just dismiss it and lock on bench numbers in a colorful graph.

One of the strengths of Rev's reviews over virtually every other review was simple: Rev plays games. This over-looked aspect is ignored by so many readers and how on Earth can a gamer take the word of non-gamer when it comes to a gaming card?

Anyone can bench but can they offer what they can see fairly?

Rev was the first in my mind to offer more than just performance numbers in a review. Too bad his review format hasn't been used more.
 
DaveBaumann said:
Well, given the background of most web reviewers, IMO, as a general rule of thumb, none of them should consider themselves any more "authoritative" as their readers. That goes doubly so for B3D given the level of knowledge and experience many of our readers have.
Agreed and as per what I said ("Most of such "reviewers" are no greater than the readers that read their writings").

Who's a web-reviewer? Someone that works for a website. Who's not a web-reviewer? Someone that doesn't work for a website. While being in such a fortuitous position a web-reviewer has two options when it comes to his "work" - (A) provide benchmarks and IQ studies (something Tom, Dick and Harry can do, and apparently is being done by almost all websites) or (B) go the extra mile by being more adventurous and innovative and hence, hopefully, provide the extra info lacking in (A). Both (A) and (B) are important and useful. My view however is that when you are in such a "privileged" position (someone with a website and getting numerous "free" hardware), option (B) should, ethically, have a higher priority. As usual, there is never enough time. That's what follow-up articles are for and are what I regard as more important than the original review.

When you are a good web-reviewer, you are trusted. When you are trusted, people will look at you as someone with an authoritative voice. There is nothing wrong, nor egoistic, with viewing yourself as someone with authority - as long as you never forget that what you have to say means two things : (a) the consumers will or will not spend their money on the hardware you're reviewing; and (b) the vendor will or will not make extra money based on what you have to say about their hardware.

At most times, it is simply a matter of using better choices of words - it is more politically correct to say "It would appear that X looks better than Y" when what you really want to say is "X definitely looks better than Y!" :) I, for one, would like to be in a position where I can say the latter without having to defend myself (against fanboys or not!).

Diespinnerz said:
The way to go is benchmarking at a given performance level and showing what each card is capable of IQ wise within that restraint. Still a whole lot of work so I'm not gonna hold my breath.
Will not work well because many people have different "thresholds" for what is termed "acceptable" or "desirable" performance. And then the priorities may change from one game to another (AA/low-rez for this game, NoAA/high-rez for another, etc).

I still believe that when it comes to comparing video cards, as long as screenshots are provided, "equalizing" IQ and then benchmarking is the better way to go. A reviewer can rarely go wrong by much when "equalizing" IQ, nor can he afford to go wrong by much (his credibility and that of his site's is on the line).

MrNiceGuy said:
Create an IQ self-review.

The review 'author' presents several screenshots of various IQ settings for each game benchmarked.

The user clicks on one from column A and one from column B that he feels are similar in IQ.

Then the website gives a comparison of the two the user has chosen.

The only downside to this type of equal IQ review is the fact that different cards have different sweet spots or perf cliffs, so there the review author should step in and mention these.
Nice idea. That, of course, would mean many mini-reviews within the review! :)

tamattack said:
Reverend, has your R300 review ever been published at ve3d.com? Or anywhere else, for that matter? If not then I, for one, would still be interested to see it.
Nope, not anywhere presently. In any case, it is too far out of date (finished back in October, older drivers, pre-DX9).

Fred said:
My only beef with reviewers, is they are forced into time constraints.
Who is forcing them? And why? How important is it to be "on time" with Anand, HardOCP, Tom et al? To who is it important to be "on time"?
 
Rev, I'd still love to read your review, and don't care if it's out of date..... although many are not using the DX9/Cat 3.0s, and, IQ & speed wise, they make little difference......
 
Reverend said:
I still believe that when it comes to comparing video cards, as long as screenshots are provided, "equalizing" IQ and then benchmarking is the better way to go. A reviewer can rarely go wrong by much when "equalizing" IQ, nor can he afford to go wrong by much (his credibility and that of his site's is on the line)

here is where i'd disagree.
Equalizing a subjective like IQ is pretty much impossible.
OTOH, equalizing a number like framerate is possible.
I still think it owuld be more worth to the readers to know how good each card can look at X framerate, rather than trying to equalize IQ (impossible) and then compare FPS. besides, most people dont care (and IMO, it doesnt matter) if Card"A" gets 250 FPS in Quake 3 but Card "B" only gets 200 FPS, if at a good framerate (ie, 100 FPS) card "B" is able to play with better visuals than Card "A". I think reviews right now have it pretty much backwards.
 
Reverend said:
Who's a web-reviewer? Someone that works for a website. Who's not a web-reviewer? Someone that doesn't work for a website. While being in such a fortuitous position a web-reviewer has two options when it comes to his "work" - (A) provide benchmarks and IQ studies (something Tom, Dick and Harry can do, and apparently is being done by almost all websites) or (B) go the extra mile by being more adventurous and innovative and hence, hopefully, provide the extra info lacking in (A). Both (A) and (B) are important and useful. My view however is that when you are in such a "privileged" position (someone with a website and getting numerous "free" hardware), option (B) should, ethically, have a higher priority. As usual, there is never enough time. That's what follow-up articles are for and are what I regard as more important than the original review.

When you are a good web-reviewer, you are trusted. When you are trusted, people will look at you as someone with an authoritative voice. There is nothing wrong, nor egoistic, with viewing yourself as someone with authority - as long as you never forget that what you have to say means two things : (a) the consumers will or will not spend their money on the hardware you're reviewing; and (b) the vendor will or will not make extra money based on what you have to say about their hardware.

At most times, it is simply a matter of using better choices of words - it is more politically correct to say "It would appear that X looks better than Y" when what you really want to say is "X definitely looks better than Y!" :) I, for one, would like to be in a position where I can say the latter without having to defend myself (against fanboys or not!).


First of all, Rev, I think the term "f-a-n-b-o-y(z)" needs to be removed from any intelligent vernacular. It has no meaning. Rather, it's merely a dodge that someone might use to deflect the focus of an argument he cannot, at least at the moment, answer. It's not a legitimate term, in other words, and nobody who is a bonafide "web reviewer" should be using it in a review--it degrades and erodes the assumed authority of those who wield it as a shield. And rarely does it actually cloak the ones who use the term, which is all the more reason it should never be a word used in a review, IMO. Also, never apologize for what you write (unless you make a mistake, of course), but always be able to explain and justify it.

In short, as a reviewer you may say, "X definitely looks better than Y," or "X looks a lot better than Y, I think" all you like. You are the reviewer--this is what people both expect and want you to do. People understand that the reviewer's review is his *opinion* and not necessarily Gospel truth. And they read you to get that opinion. The worst reviews possible are reviews in which no opinions are actually rendered, but rather the reviewer engages in a lot of equivocating, instead. It often becomes obvious in such a review that the reviewer does have a favorite, does have a preference, but is *afraid to say so.* This always diminishes the power and force of the review and in short simply makes it much less than it could be, and can often actually pervert what it was the reviewer was actually trying to get across. It's not a reviewer's job to be "politically correct"--it's a reviewer's job to review. That's why I read them and what I look for. I think trying to be diplomatic because of concern that some might not think you diplomatic is a waste of time because no matter what you say some will think that anyway.


...Will not work well because many people have different "thresholds" for what is termed "acceptable" or "desirable" performance. And then the priorities may change from one game to another (AA/low-rez for this game, NoAA/high-rez for another, etc).

I still believe that when it comes to comparing video cards, as long as screenshots are provided, "equalizing" IQ and then benchmarking is the better way to go. A reviewer can rarely go wrong by much when "equalizing" IQ, nor can he afford to go wrong by much (his credibility and that of his site's is on the line).

Provided such "equalization" is possible. Sometimes it is, sometimes it isn't. When one product has obviously better image quality than another, to glaze over that observation in the interests of "equality" is to actually provide a review picture which the reviewer himself thinks is dishonest, and to shortchange both the review and the reviewer.

If you mean "equalize" in terms of resolution, FSAA, AF, etc. ad infinitum--I absolutely agree with you. If this stops you from commenting on which product you think is better in terms of IQ, once those conditions have been achieved, then I have to completely disagree. Again, the reader is not interested in a diplomatic display--he's reading your review because he wants to know what you *think*--he's not interested in what you think you are "supposed to say," instead.

Of course, in cases where you honestly perceive no difference in IQ, it is eminently acceptable to say so (obviously.)

The only downside to this type of equal IQ review is the fact that different cards have different sweet spots or perf cliffs, so there the review author should step in and mention these.
Nice idea. That, of course, would mean many mini-reviews within the review! :)

How so? It may broaden the scope a bit, but would not change the review subject. But basically I think this can be avoided by the reviewer simply showing what he thinks are the "sweet spots" for both products compared. If you are doing a comparison, I think a reviewer should always keep in mind that what he's actually doing is reviewing all the products he is comparing--instead of just one of them. Years ago I remember one thing I saw more than one reviewer doing which used to actually make me angry at the time (sort of)--they'd take a card that was the subject of the review, over clock it, and then compare it to a stock-clocked competitor card. Then on the basis of the overclocked FPS scores of the one card contrasted to the non-overclocked scores of the other--they'd proclaim that the overclocked card was indeed the winner of the performance race between the two products compared! I saw this several times between a 3dfx V3 and a TNT2, if I recall correctly. Unbelievable, right? But it definitely happened more than once. So your idea about "equalizing" things in a comparison as much as possible is a good one.

Who is forcing them? And why? How important is it to be "on time" with Anand, HardOCP, Tom et al? To who is it important to be "on time"?

It's not important to be "on time"--such as "first," etc., because no one alive today can tell you who wrote and published the first 9700P review--and nobody cares....;) But it is important to be "timely" which is why I can respect your decision not to run your own 9700P review.

If I had a singlemost gripe about reviews it would be that they are too often less than candid. I read reviews because I am sincerely interested in hearing what a reviewer thinks about a product, and I think a good review is also a thorough review (I have no taste for single-pagers.) I want to hear as much negative--if it is there--as I do positive. That doesn't mean I will agree with the reviewer--but I'd still like to hear what he *doesn't like* as much as what he does. Most of the time we get the latter but never the former. With some products this is really justified because some of them are actually very good. But what happens far too often is that reviewers are simply *afraid* to tell us what they don't like--I've not determined whether that stems from a fear of reprisal from the manufacturer, or a fear of rejection by the intended audience. Sugar-coating everything is the safest course, and the one that most reviewers take.

For instance, how often to do we hear:

"Well, I had a few driver problems here and there but nothing that prevented the review from proceeding and I am confident that company Y will fix these bugs in an upcoming driver release. In fact, I just emailed the company about it and have their assurances, so all is just Jim Dandy and Peachy Keen."

What would be nice is if the reviewer would detail what his problems were along with the manufacturer's assurances--and let the readers decide whether these were problems they felt would be taken care of in future driver/firmware updates to the products. In fact, along with any detailed critical analysis of any problems present with the products at the time of review, I think it would be great if reviewers would grant the manufacturers space to answer the criticisms within the body of the review--again letting the reader decide whether the manufacturer's explanation is credible. But I doubt something like this will ever happen, right? *chuckle*

Edits: Typo
 
DemoCoder-
What you really want to benchmark is not IQ and not performance but IQ / Performance. Of course, that assumes IQ could be "quantified", which it can't.

This surely wasnt the case with the typical GF3 vs 8500 generation of "shootouts"- nobody seemed to care about using any form of "balanced" IQ, regardless of the techniques in use. (i.e. SS vs MS, or bilinear vs. trilinear AF). In fact, it only became a matter of comparing one set unfairly (and silently) as long as the performance lead was held by one IHV, followed by a 5 page in-depth image analysis for the portion where the lead wasn't substantial or on the losing side. So it just comes down to the highest bidder, as always.

This will always be the case when any kind of enhancement, be it performance or image quality, can be discounted only when in favor of a particular IHV. If there is a measurable performance lead, this will be discounted as "great, 250 fps in Counterstrike" or if there is a measurable quality lead it will be "there is no difference between 2x, 4x or 6x that your average Joe walking down the street will notice on a 13" black and white monitor using composite video TV-out feed" kind of comparison.
 
No, the problem is that IQ isn't quantifable. Person A zooms in on a FSAA screenshot and notices some small artifact between two IHV implementations and proclaims one better. Person B zooms in AF screenshot and proclaims the opposite. Person A discounts some of the AA artifacts for the speed boost you get in that mode. Person B discounts some of the AF artifacts because of the speed boost you get in that mode. And Person C altogether is a person who might be willing to trade IQ for speed. Person D could be a person who trades speed for IQ.


The problem with the equalization idea is that you can't objectively equalize two subjective quantities. It's not scientific. As soon as you equalize AA IQ, someone else will yell to equalize AF IQ, etc. You may as well talk about equalizing Rembrandt and Van Gogh.


Perhaps the other suggestion is better: equalize the framerates, compare the IQ. E.g. @ 1600x1200 @ ~60fps, whose got best IQ.
 
Status
Not open for further replies.
Back
Top