How benchmark 3D Cards, today

Ya.. how to stick to topic today.
me- pick 5 or so of the latest (good) games which stress modern hardware and have some nice shadermark 2.1 benchmarks and some other synthetic benchmarks (not 3dmark 06).
Then check out IQ on a perframe basic for atleast 1 hour of game play.. maybe they only cheat on certain frames :D
er.. I mean play some games on the card carefully exampling IQ at default settings and with all opts disabled.
A nice texture filtering app would be good also, like what beyond3d uses.
 
I think that a reviewer have to ask himself

"what the consumer want from a review?"

I believe that a consumer want this things:

a) what 3d hardware is better with actual game?
-> what settings make me play with MINIMUM framerate above 30 fps with common heavy-demand games?

b) what 3d hardware is better to play with eye-candy mode?
-> I don't care about costs, I want to know whic card give me the best IQ with all settings maxxed

c) what 3d hardware will mantein his value in future?
-> I care barely of actual games, I'm buying a new pc and I will stay with this card for at least 2 year, please benchmark it with worse situations possibile, so I can compare to others card and make my choice

when I talk about 3d hardware, I talk about 3d hardware, no software tricks, no drivers 'optimizations', no cheats
the same condition for all the cards tested.

another point is that, if ATI R520-R580 have no hit, from 4xMSAA / 8xAF to 6xMSAA / 16xAF is unfair don't benchark this, only because with nvidia any AA over 4x will produce an incredible hit
if a card has a value, a good point, the reviewer have to underline this, in order to make a good service to his reader


at now is still possibile do this, jumping from a review site to another, but no one yet give all the info required from consumers
my best fair, detailed prefered bench-review sites are hardware-fr.com and beyond3d.com
 
RedBlackDevil said:
How make usefull posts, today :rolleyes:
Click to expand...


Okay, here's how I would test graphics cards: I'd make it a double blind test, as follows:

6 Graphics cards (2 value cards, 2 mainstream cards, 2 high end cards, 1 each from ati, 1 from Nvidia)
D6 dice
1 piece of paper, envelope and a pencil
6 assistants to test the cards, each selecting a number from 1 to 6
6 identical computers (other than gpus) or a whole lot more of time and one computer
1 assistant to roll the dice, each card is labelled with a number.
1 assistant for changing the settings.

-The dice rolling assistant will not know numbers of the 6 assistants (who is who). He will only know which assistant number got which card, the assistant will never meet the others to tell them which card they got. Instead the results of dice rolling will be written at this time to an envelope which gets sealed.

-The assistants testing the cards will never know which card they got. They will not be allowed to touch the control panels so they won't even know if they had an nvidia card or an Ati card.

-Settings changing assistant will not meet anyone of the testers, the tester assistants will leave the room during settings change and return once complete and the settings changer has left the room.

The testers will continue testing the card until they have reached their conclusion about it, write notes, and then will increase their number by one, therefore getting a new card to test. In other words, every tester will test every card with a number of games and settings, not knowing which card they used at which time.

In the end, after all testers have tested all cards and formed their conclusions about it, an article will be written to a website, the writer of the article only knows the cards and assistants by number.

After that, the dice rolling assistant will give the writer of the article the envelope containing information about which card was which and that information will be added to the end of the article or throughout the article for clarity.

What do you think? Quite an undertaking, no? No bias at least!
 
Last edited by a moderator:
I'm perfectly happy with the current testing procedures combined over various sites. When you check out the reviews on B3D, [H], Anand, Hexus, ixbt and Guru3D you'll get a good enough picture. Obviosly, not so from a single review on a single site.

I wish people (especially various f-boys) would just stop bitching, that would make it much better already.

EDIT: and you'll definitely NEVER be able to satisfy everyone, so why even bother?
 
Mendel said:
Okay, here's how I would test graphics cards: I'd make it a double blind test, as follows:
Click to expand...
Well, the assistant who is to change the settings has to be competent and unbiased, since he would be able to manipulate the results.

Other than that: Sounds great. Now you just need the manpower and time to do it :)
 
_xxx_ said:
I'm perfectly happy with the current testing procedures combined over various sites. When you check out the reviews on B3D, [H], Anand, Hexus, ixbt and Guru3D you'll get a good enough picture. Obviosly, not so from a single review on a single site.

I wish people (especially various f-boys) would just stop bitching, that would make it much better already.

EDIT: and you'll definitely NEVER be able to satisfy everyone, so why even bother?
Click to expand...

if noone bothered there would be no progress, good to see people trying to get better
icon14.gif
 
RedBlackDevil said:
whay you are so hostile?
if you are good with a 3-4 year ago benchmark style, ok
but if we are talking, is to improve this, and years in technology is hundreds years :LOL:
Click to expand...

No hostility here, where do you get that from?

We compare cards for speed in a buttload of games/apps, we do compare them for IQ. What more can/will you do? It takes no Einstein to figure out that there are not many more options left.

EDIT LOL@ "3-4 years old benchmark style". You DO know what the word benchmark means?
 
geo said:
I would think that's fairly obvious --because a signficiant portion of the community is unhappy with the status quo! Now, whether they can ever be made happy, or at least if they can be made happy without pissing off a different portion of the community, is an entirely different question! :p
Click to expand...

But again the question: what options do you have for benchmarking besides comparing speed over a (huge) variety of games/apps and IQ-comparisons?

I can't think of any, maybe a comparison of "warm, fuzzy feeling"-ness or the colours of cards, dunno :p
 
I'd like to see a range of cpus in *every* review. For every guy who's itching to have a new gpu and has an FX60 to pair with it, there are five or more guys who are itching for a new gpu but have something significantly less exalted to pair it with.

I roll my eyes every time I see a mid-range or value gpu paired with the latest/greatest cpu for benching. It's not that that isn't interesting, as one data point, but it's almost certainly a hell of a lot less interesting to the potential market that card is aimed at than something more mid-rangy, and even a cpu that is bordering on "oldy/moldy" status!

Did I just suggest creating a shit-load more work for the reviewers? Did I just piss off IHV PR with the idea of showing their new babies in "less than optimum" platforms? Yeah, well, cry me a river! :p If the subject is what would actually be useful in improving reviews for, y'know, real consumers, then there ya go.
 
CPU-variety is a very good point indeed.

EDIT: didn't [H] and Anand test with one high-end and one lower-end CPU some 2-3 years ago?
 
Last edited by a moderator:
geo said:
I'd like to see a range of cpus in *every* review. For every guy who's itching to have a new gpu and has an FX60 to pair with it, there are five or more guys who are itching for a new gpu but have something significantly less exalted to pair it with.

I roll my eyes every time I see a mid-range or value gpu paired with the latest/greatest cpu for benching. It's not that that isn't interesting, as one data point, but it's almost certainly a hell of a lot less interesting to the potential market that card is aimed at than something more mid-rangy, and even a cpu that is bordering on "oldy/moldy" status!

Did I just suggest creating a shit-load more work for the reviewers? Did I just piss off IHV PR with the idea of showing their new babies in "less than optimum" platforms? Yeah, well, cry me a river! :p If the subject is what would actually be useful in improving reviews for, y'know, real consumers, then there ya go.
Click to expand...

We have a CPU / Video Card scaling article that is nearing completeion now. 6 different CPUs, 3 from Intel, 3 from AMD. Dual core included in both. 6 or 7 video cards used. Only used NVIDIA. BFGTech sponsored the article. We stuck with one brand as it is my opnion that it will give us results that are easier to represent without a bunch of argument on the video card end. We have used real-world gameplay and IQ scaling as the measure for comparison as we do in current video card evaluations.

This has been a tremendous amount of work and required a lot of time and money to get done.
 
But I can imagine you can put 2-3 "standard" systems together with low, mid and high-end CPU respectively for video card tests? You'd only have to change them 'bout twice a year.

Right now, a sytem with AthlonXP 2500+, one with Athlon64 3800+ and one with the current top CPU would do the job fine methinks. So the budget people drooling over the new GFX-card would know if it makes sense putting it into his system.

I know it would be a bit of work, but it would be nice.

Kyle, didn't you do something like that when you tested Doom3 back then?

EDIT: the search function on [H] is truly abysmal, there's some potential for serious improvements there ;)
 
Last edited by a moderator:
FrgMstr said:
This has been a tremendous amount of work and required a lot of time and money to get done.
Click to expand...

I don't doubt it. Resource realites are resource realities. Doesn't change the answer to the question tho.

Probably a periodic visiting, such as the article you're pointing at, is a reasonable compromise. It'd be nice to see more sites doing it more often. If several sites were doing such once a year it'd probably mostly cover the need. Tho on "release day" there is still the danger that old Joe is liable to succumb to his itch and run off to Newegg for a gpu that isn't going to improve his quality of gameplay as much as he's thinking it will due to the cpu he has to play it with.
 
Mendel said:
Okay, here's how I would test graphics cards: I'd make it a double blind test, as follows:

6 Graphics cards (2 value cards, 2 mainstream cards, 2 high end cards, 1 each from ati, 1 from Nvidia)
D6 dice
1 piece of paper, envelope and a pencil
6 assistants to test the cards, each selecting a number from 1 to 6
6 identical computers (other than gpus) or a whole lot more of time and one computer
1 assistant to roll the dice, each card is labelled with a number.
1 assistant for changing the settings.

-The dice rolling assistant will not know numbers of the 6 assistants (who is who). He will only know which assistant number got which card, the assistant will never meet the others to tell them which card they got. Instead the results of dice rolling will be written at this time to an envelope which gets sealed.

-The assistants testing the cards will never know which card they got. They will not be allowed to touch the control panels so they won't even know if they had an nvidia card or an Ati card.

-Settings changing assistant will not meet anyone of the testers, the tester assistants will leave the room during settings change and return once complete and the settings changer has left the room.

The testers will continue testing the card until they have reached their conclusion about it, write notes, and then will increase their number by one, therefore getting a new card to test. In other words, every tester will test every card with a number of games and settings, not knowing which card they used at which time.

In the end, after all testers have tested all cards and formed their conclusions about it, an article will be written to a website, the writer of the article only knows the cards and assistants by number.

After that, the dice rolling assistant will give the writer of the article the envelope containing information about which card was which and that information will be added to the end of the article or throughout the article for clarity.

What do you think? Quite an undertaking, no? No bias at least!
Click to expand...
Sounds Fun!!!
You should tr tp pitch it as a hit collector...it night even spurr a whole new breed of review method using blind and double blind tests!
Any takers?
 
_xxx_ said:
But I can imagine you can put 2-3 "standard" systems together with low, mid and high-end CPU respectively for video card tests? You'd only have to change them 'bout twice a year.

Right now, a sytem with AthlonXP 2500+, one with Athlon64 3800+ and one with the current top CPU would do the job fine methinks. So the budget people drooling over the new GFX-card would know if it makes sense putting it into his system.

I know it would be a bit of work, but it would be nice.
Click to expand...

I'd settle for *one* game in the current suite (tho preferably the most demanding) tested that way on each new gpu review. Provide at least a quick thumbnail for Joe Enthusiast to ponder before he plonks down for a new gpu to go with an older cpu/mobo.

Tho I also think it's pretty silly to be testing mid-range/value gpus with the current top cpu anyway. I might even reverse the above --test it for the entire suite with something more mid-rangy cpu-wise, and provide one further test with the most demanding game with the burliest cpu you can get your hands on.
 
You must log in or register to reply here.

Similar threads

A
Holy Wall of Text: My impressions post Playstation 4 (PS4) unveil
2 3 4
Replies
70
Views
20K
(((interference)))
I
2
is all of this possible?
2
Replies
22
Views
8K
Simon F
Simon F
2
  • Locked
ms/nv/ati=bad iq for games; ?'s
2 3
Replies
45
Views
9K
Rys
Rys
A
  • Locked
List of Rendering Resolutions
Replies
4
Views
1M
AlNom
A
N
16x supersampling (4xSS x 4xMS) amazing
2
Replies
21
Views
7K
Moloch
M
Back
Top