How benchmark 3D Cards, today

I am sorry our vide card content has no value for you. Certainly there are many other sites that we link on a daily basis that show different ways of doing things. Hopefully one of them in right for you.

I bought my HUMMER H2 about 3 years ago and use it for lots of off-roading trips that my wife and two children usually accompany me on. It is comfortable for the entire tribe on long trips and quite capable when it leaves the pavement. After the small business tax breaks that were exploitable, I paid less than US$33,000 for the vehicle...hardly and exotic by any stretch of the imagination and the normal price for a SUV here in the States.

I am unsure why money has anything to do with your argument on how we evaluate video cards. Without HardOCP making money, we would not be able to pay editors like Brent Justice that spend countless hours testing real-world gameplay, the way we see it best done. Those same profit dollars just allowed us to bring on two more video card editors that will be focused more on mid and low level video cards, to help bring more mainstream content. I try my best to make the best living I can using honest and moral means to support my family. I am not poor, but I am far from rich.

So to answer your question, no, the HUMMER does not fall under legal fees. My wife's transportation is a 2001 Dodge Durango (Got rid of my 1997 Jeep to better fit the expanding family.) if you are wondering and it was not purchased as part of any legal fees either.

 

You are forgiven
33 grand for an suv isn't that bad but I can think of a better way to transport the family which doesn't involve paying up the ass for gss.
My sister has a durango, so far it makes a wierd noise when taking off making turns some times kinda feels like the tires are spining but without the tires spinning.

The problem I have with [H] se the very mainstream approach- it just isn't technical enough for me.
For videocards sure it shows you want your review staff thinks it the best playable settings but it would be better if you showed how it performed at other resolutions/settings and how you hate"fake" (synthetic) benchmarks despite not knowing why you would want to run those "fake" benchmarks.
Like for the R580 for example, it would be nice if you had a pure pixel shader benchmark to show why it's faster than the R520 in a number of games, and how much better it's performing purely in pixelshader speed since that's the only advantage it has over the R520.

I also have a problem (like many) in the way you conducted your 1000 series review/preview.
Ya know the whole we don't review cards that aren't really released.
politics...

Btw you may claim you aren't rich but like the president, you have access to hardware that you would either have to rich or stupid to have.

 

Though I am all for seeing how far you can push the latest cards at max qualities while gameplay remains playable, I think that high end cards, and maybe even mid-range should also be tested at the two common widescreen LCD resolutions at whatever settings are playable.
I do support viewing max playable settings, but having LCD resoltions with variable settings might be useful as gamers who have ditched their CRTs have diferent issues when going to lower resolutions.

 

We are expanding to widescreen high-end gaming now. Our new mid and low-end evaluations will be more specific to the resolutions we think they would be more fitting to. When it comes down to it, it is just a terrible amount of work to cover every conceivable perspective, and sometimes we at HardOCP just have to make some decisions on how we are going to cover it. We will get better though, promise.

 

optimizations

personally I'm all for benchmarking with optimizations on. If the graphics look as good, then leave them on. You can compile programs specifically optimized for intel, amd, via, or other cpus, and to take advantage of sse and mmx. Would you turn these optimizations off because the other side isn't using them? If ATI or Nvidia is willing to optimize shaders for peak performance on their products, why not let them? I think it's retarded that people complain about this. It shows a total lack of understanding of computer hardware and software. Of course ATI and Nvidia know how to write code for their video cards better than anybody else, so let them and learn from them.

I am all for ditching 3dmark, because the optimizations there are largely wasted. I say benchmark as many real games as possible so ATI and Nvidia feel pressured to make all games run well on their cards.

 

Apparently you're either blind or don't own an nvidia card.
Since [h] is supposed to be about gaming, how many people with nvidia cards game with them on quality setttings?
The first thing I did when I got my 6600GT was turn the opts off, same with my 7800GT, except the 7800GT has worse filtering
I can't stand the IQ with the opts on and even off they could very well could stand to improve.

The difference between using SSE is that using SSE doesn't degrade the quality, it speeds it up with no loss in "IQ" so why not use SSE1/2/3?
It speeds things up with no downsides if coded properly.
Nvidias opts are very visable.

Btw that's a bad first post since you revealed you either have no experience with graphics card or need glasses.

 

Plenty of other sites for ya then. I like to think we offer something other sites don't, therefore read several sites and draw your own conclusions. Our method has real value if you want to know what the gaming experience is like. But if all you want to know are theoretical shader numbers, we may not be what you are looking for

 

That's the oddest response to our evaluation method I've heard yet.

You actually think using games is a bad thing? Why do people buy the latest 3D graphics cards you think?

Hrm, I think the answer might start with a "g" and end with an "ames"

I could be wrong though, let me check the math on that...

 

I'd like to atleast know how the graphic card scales with resolution and fsaa.

Btw a certain unnamed site does the same thing as you except they don't post fps numbers.

How do you offer something other sites don't by just posting the results for one res you deem playable?

And how is a pureshader benchmark a bad thing?
Are you going to tell me that you don't find it interesting to see what the 48 alus on the R580 bring over the R520 in pure shaderbenchmarks?
We're talking 16 alus vs 48 alus!

 

But I think that's the point. In theory, X1900 brings x times performance to the table, in practice it's a bit faster then the X1800 in most apps, and a lot faster in very few. If I assumed that the FPS charts here at B3d show what I could do at home I'd be very dissapointed. I don't think that just running demos and scaling resolutions provides enough data. I would've loved if Dave or whoever is reviewing could actually comment on how HE would use this card, and what settings he finds comfortable. Sure, it adds a very subjective feel to the review, but at the end of it, you'd know how another gamer would utilise the card.
All of the Time Demos, and theoretical benchmarks add up to one thing- technical superiority. That doesn't really indicate real world use, if I don't kow what the gamepley feels like, what good does it do me? Ok, an example is the recent Nvidia 7800, everyone says it's filtering sucks, did that show on any technical benchmarks here?
I don't think that techinical reviews should be discarded, but subjective analysis should be added. Comments on how it feels to use the product.
Not many people buy these cards in order to run shadermark, and if so it's only in the first few days (unless they have OCD and E-Penis syndrome). Pesonally speaking, when I got my 9700 Pro I benchmarked it to death. By the time I got my 6800GT I didn't benchmark it at all. The closest thing I did was run it in all my games to see what settings are playable, and stress test it with 3dmark to see it's stable.

 

Well it is in fact much faster than the R520 in a few games like AOE3 for example.
Not wanting to see the theoretical shader performance of a part with over double the alus is just plain lazy imo.
Shader benchmarks also show why the NV3X sucks in DX9 games.
Using those shaderbenchmark you can predict how the NV3X plays DX9 games... very badly

 

I am not overly fond of how [H] bench but I am glad they are doing it because they add variety to the type of reviews out there; no doubt somebody hates what I like in benches and they might be better catered for looking at the [H] review.

I tend to look at a whole lot of reviews and just get a feel for the card rather than the nitty grtty. For instance with the new X1900 ( single and in crossfire ) it seems that as the AA goes up you're much better going for that than a 7800, also that a single would do most of the time and crossfire is sometimes overkill or has no improvement. I could not tell you individual results except for the fact that in FEAR it does pull out a big lead so suggesting the brute force shader power is starting to show it's metal.

 

The More the Better

More information/accuracy is better than less.

 

Maybe a little more?

Was that too little information to give any helpful feedback or insight?

Maybe more would help, but might be too much (not unusual for me); but if it's too much you can ignore or skip to the good bits, unlike discussing this with a friend where you have to listen to the whole thing even if you're not interested.

The more raw information then the user/reader can decide what to value and what to reject. Having the reviewer simply give more conclusions (one for each games) just takes that out of our hands and for some does what they were doing in their head anyways. [H]'s hitograms should tell you if the resolution has major issues/drops and whether to you that's playable levels. However reviewer A saying it's fine to them, doesn't tell you if jump fragging while going a 180degree turn is fluid enough in Quake4 versus creeping along molasses slow in SplinterCell3. Both require different levels of 'acceptable speed' IMO.

Also the methods used to ensure that that information respresents a true apples to apples comparison is needed, and as has been said IQ comparisons like using compresonator seem necessary though texture crawl is hard to show snas video. Think of all the people who harp on 'good optimizations' but at the same time say that a test of the two cards at 'quality' or 'HQ' would be ok, despite now showing the actual difference between the two. Optimizatons on an off should be the standard for comparison, and the effort must be made to compare the optimizations off to show that IQ is indeed equal or else the review must say there is one that obviously is better/worse (even if only in stills) as we're talking about comparisons. One company may have great optimizatoins that don't change IQ and while it does less work the overall result is the same as card B. Well that's OK for part of the test, but the optimizations off results give you an idea of raw performance and how a card may react to a new game before driver updates which may take 1-3+months after a game release (usually depending on popularity before and after launch). Which brings up another point, more non-standard games (like Evil Dead Regeneration) and demos (like Farb Rausch, Rthdribl) would help expose this raw handling of unfamiliar or unoptimized code.
It would also be nice at some point, to give some insight into strengths and weaknesses on a per component basis to help give an idea of how this card might handle future apps/games (for this the individual tests and not overall bungholios of various 3Dmarks, rightmark, shadermark, etc are good).

The problem with the mid/low-end is not just matching them with similar systems like you did in the GF6600DDR2 review, but also seeing how much even they may be system limited (tell you if there is ANY future in those choices) . Putting an X1300S or GF6600 on an FX60 with 2GB of OCZPC4000 with a 150GB raptor doesn't always make sense but it does for people trying to contemplate their upgrades who might get little benifit from that jump to an X1900, but the move to that config from their 3200+ with 512MB of generic PC3200 might be enough to make their rig playable more playable with reduced IQ, while just upgrading the graphics card may be only mildly better than before, but will allow you to turn on AA, unlike before.
For a graphics card review that may be tough to do because benchmarks must be re-run after re-install, etc, but if you want to go 'lean' on the information being provided for results, then invest that obvious time saving into giving us more information off that subjective perspective. Tell us if the graphics boost matter as much as the system boost even for such system, instead of wasting your/our time benching a GF6600 in FEAR @ 1920x1200/1440 with No/2X/4X AA.

The most important thing in any review, be it subjective/qualitative playability or objective/quantitative FRAPS results, is that all things must be equal, especially output IQ. That that hasn't always been the case invalidates past reviews and calls into question the subject claims of equality by those reviewers. That you say it's equal or good enough doesn't mean anyone else will, just take the CRT vs LCD for gaming as a perfect example, so some basic proof is good.

And what it comes down to is, the more information, the more reliable, and the more applicable to each person "what's in it for me", the better. Narrow down what you provide and you narrow the audience that can use the information for whatever use.

For some people it's too much information, like this post, but they'll just skip / skim-through it anyways so they too are covered.

Just my two frames' worth as always.

 

So what's the hard form of reviewing?

1. Record a walk through and play it back using various settings until the "playable" setting is found?

2. Play through uniquely each time with various settings until "playable" settings are acheived?

 

You got me wrong about it. I said that the best games for benchmarks are less known games. There are many (3D) games out there that support SM 3 and DX9 but they are not that famous like COD II or FarCry.

 

These optimizations have big image quality penalty.

 

It's not about [H]; it’s about everyone.

 

My thinking is that there's no single, good way of benchmarking or testing a video card, which is exactly the reason why nobody should base their judgement on a single review or a single site's testing. Broadly speaking, there's three ways to review a video card:

1. Technical analysis (a la Beyond 3D)
2. Subjective analysis (a la bit-tech)
3. 'Traditional' analysis, using timedemos and FRAPS (a la... well... Elite Bastards )

There are up sides and down sides to all of these methods, but taken as a triumverate they cover pretty much every aspect of a board and architecture. No one site has time to cover all of these aspects in the real world, thus read a review from each of these three 'disciplines' and I would say you're likely to end up with a pretty good grounding in what's good and what's bad about a particular board.

Interestingly, I think discussions of image quality is the one thing that has suffered review-wise recently (and I stand guilty of this too!) - Perhaps this is because we've reached a point where everybody knows broadly where ATI and NVIDIA currently on this front, but it still seems to get very little attention from many quarters. For example, when I reviewed the X1900 XTX on R580's launch day, someone commented to me that it was the first time they'd seen an in-game screenshot comparing ATI's standard and rotationally invariant anisotropic filtering!

Anyhow, I guess what I'm trying to say is that perhaps there shouldn't be any real focus on 'one testing method to rule them all' - The variety of reviews available around the web now is what makes numerous sites out there a valuable resource, and in that sense I think we're in a much better place now than we perhaps were pre-R300 and 3DMark03.

 

what is important to know from a benchmark is

How much the card are good when it came to eye-candy mode (maxx AA)?
How settings make it play at 30 or 60 fps in the real world?
how the cards will perform with future high-demand games? [hardware stress]