HardOCP - Cheating the Cheaters

[Lezmaka]

There is something to be said for doing a review this way, compared to how others compare cards, assuming it's "done right" of course.

If your intention for a review is to show people how a card performs in their favorite games (as opposed to which one is the best overall), "doing it right" means using a lot of games from different genres, and not just brand new games either. Just because I buy a new video card doesn't mean I'm going to stop playing games I used to play with the old card.

I would also say this would require a IQ comparison for each game used. Don't just use one or two games and assume IQ is OK for all the others.

If I want to know how cards perform in the games I play, I would look for a review like this. If I want to know the technical stuff, I come here. And as any consumer should, I wouldn't make my decision based on 1 or 2 reviews, whether it's a pure gameplay review or a pure synthetic review. I would read everything I can find, weigh everything and then make my decision.

Also, you shouldn't just go by sales data and whatever the "latest games" are either. Load up GameSpy, All Seeing Eye, or even Gamers Search 2.5 and see what games people are actually playing (no matter how old). HL (and all its mods) is still the most popular game and if I played it, I would want to know how it would perform and look on a card before I buy. Sure, it's 20 years old, and any card coming out, including mainstream cards, should be able to max out everything. Buf if you really want to do a gameplay review, you should include games like this along with an IQ comparison for it. More examples are Unreal Tournament, Quake3. As of right now, there are more than 3000 people playing each one of those games (according to Gamers Search 2.5) and only around 1400 are playing UT2003, yet I would have to guess that UT2003 has been used in reviews much more than the others.

I guess what I'm saying is that if you really want to do a gameplay review (as a service to gamers, not just to see which card is the best), use games that people are actually still playing, no matter how old or obsolete they are, in addition to the top selling and new games, and include an IQ comparison for each game.

 

[THe_KELRaTH]

UT1 is brilliant with a R3x0 cards; the Hi res textures update, frames fast enough to have Vsync on, AA/AF at max and the only game I make use of Trueform and it still plays better than UT2004 (none of the fisheye effect when you rotate).

 

[WaltC]

Obviously, games themselves can be optimized for or "cheated" in as easily as synthetic benchmarks can be manipulated. What has always bothered me about this approach as I've seen it reiterated by various [H] personnel over the last year, is the "either-or" nature of the approach.

There's absolutely no reason you can't run synthetic benchmarks and do all of the "real-world" testing you want to do, is there? I mean, no reason at all to state that "if we do real-world game testing we cannot do synthetic benchmark testing," is there?

And so, good and thorough hardware reviews will endeavor to do both, of course, and exclude neither. So I think that should be the litmus test moving forward: if you don't do both, you don't rate.

I always thought [H]'s approach to the various nV3x/Forcenator cheating events that took place last year was akin to "throwing out the baby with the bathwater"... I never could figure out why, instead of blaming the benchmark software, the IHV wasn't blamed--seemed far more logical to me throw out nVidia instead of 3dMK... But that's just me...

I just don't buy the theory that IHVs are mindless sin-slaves who *must* cheat benchmark software "because it's there" any more than I blame a bank because it is robbed, or the homeowner whose home gets burglarized. I think the sentiment is entirely misplaced. The culprit is the criminal, not the victim.

If I could write the rules, I would write this one:

"Cheating IHVs get dumped when they get caught, and we don't blame the software for the IHVs who cheat it, unless we can prove the software is in complicity with the cheater."

I have to say that I think "We don't do synthetic benchmarks anymore because nVidia cheated them," to be a fairly hypocritical, not to mention impractical, approach, and it simply won't solve a thing. What's going to happen, for instance, in the event "real world game testing" for Doom3 reveals that nVidia is cheating in some fashion? Are you going to throw out Doom3, as well? What about all of the other real-world games in which it becomes apparent that something is being misrepresented?

Frgmstr, how about UT2K3, when it became apparent nVidia wasn't really doing trilinear filtering on detail textures at all--even though nVidia said it was? Did you throw out UT2K3? If not, and I don't think you did, it's difficult to see any difference at all there with some of the things nVidia did with 3dMk03. But you're throwing out the synthetics whole-hog as a result.

Bottom line is that if you blame the software instead of the IHVs who cheat the software, you might wind up doing 10,000% more work just trying to find some software somewhere that some IHV isn't cheating on in some fashion... Seems much more efficient just to throw out the offending IHVs until they clean up their acts--don't you think?

 

[Eronarn]

No more apples to apples?

I like it. I like it a lot.

 

[Quitch]

UT1 is brilliant with a R3x0 cards; the Hi res textures update, frames fast enough to have Vsync on, AA/AF at max and the only game I make use of Trueform and it still plays better than UT2004 (none of the fisheye effect when you rotate).

I've always found that enabling High Detail textures reduces my Radeon 9800 Pro to a crawl under both Unreal and Unreal Tournament.

 

[sethk]

I've read through a lot of the thread and I will summarize my feelings about the [H] reviews thus:

I feel less informed reading them. They feel rushed, incomplete and less informative than, say the Beyond3D review.

I've been a [H] reader for a long time, and a [H] forum member for a long time (under a different name) but at this point the reviews are starting to look like a snapshot in time. It's a way of saying, as of xx/xx/xxxx videcard xyz got f frames per second in this game. That's not a review. Sure there was some attempt to fill this out a little bit, but I came away feeling that this was very incomplete, and I could practically taste the bile that [H] seems to have, as a site, towards certain or maybe all hardware manufacturers and their corporate machines. I can't say I blame him / them given the history, but it's unprofessional, bitter and just doesn't make a good read. Don't alienate your readers on top of everything, I don't know what you're trying to accomplish.

That's the obligatory 2 cents from a former [H] fan who hopes things will improve again.

 

[Ichneumon]

Kyle has taken a whole lot of shit the last year or so --and certainly some of it he brought on himself either by not explaining well enough or getting too defensive about what he was trying to do. In this world when you try to turn the battleship there will be a lot of resistance. Having said that, looking back over that period, and seeing that there was a bit of flailing about, it seems to me that he has been remarkably consistent in what it is he is saying and the ultimate goal he is trying to achieve.

I may not entirely agree, but it seems those who've wanted to pigeon-hole him as the plaything of this IHV or that in the exigencies of the moment should look at the bigger picture over time and see he's been saying the same things no matter whose ox is getting gored today.

Heh... only because he was a total and complete sucker to Nvidias marketing right up until the point that they smacked him right in the face in front of the whole world when they made him into a lier. So many others saw what Nvidia was doing to him long before he did... He fought tooth and nail to say it wasn't so... and he paid for it.

Now he's once again revising history.

 

[cthellis42]

The more tools in operation, the better.
The more people spending more time evaluating them, the better.
We get more information, and more minds bouncing off each other as to what's going on, and in the end we get a much more complete picture.
There is no "easy invalidation" of anything--just recognition of the proper way to use the tools available.

Of course there is a limited budget--both in time and money--for the people that do the reviews, so we have to expect that the choices of where and how to focus from one person to another will be different.

...and in a way, that helps us too. It's "more information" coming from more and different styles of how to process the data coming in, and their different ways of presenting it. (It just makes it harder for US to compile it all together. )

 

[Dave Baumann]

I've read through a lot of the thread and I will summarize my feelings about the [H] reviews thus:

I feel less informed reading them. They feel rushed, incomplete and less informative than, say the Beyond3D review.

I'll wager that Brent spends a lot of time on his reviews, almost certainly moreso than I can. However, we are now clearly answering different needs in the "market" and as such they are not really comparable, but there's no reason why they can't be complementry. I'd never advise someone to base a purchase solely off a B3D review - its purpose is to provide some understanding on the product, but I'd always advocate the gamer looks to a wider range of reviews (sometimes the picture may be a little more cut and dried for the coder).

Personally I applaud Kyle and Brent if they are bold enough to do this - I wouldn't think its going to be easy, but I think it could be worthwhile.

 

[digitalwanderer]

However, we are now clearly answering different needs in the "market" and as such they are not really comparable, but there's no reason why they can't be complementry.

Actually I think Brent and your x800 review were about the two most complimentary reviews I ever read.

Yours hit all the technical aspects fully and explored the potentials of them, Brent's covered the (gag!) "real-world" gaming experience of each card.

I thought 'tween the two of them I got a pretty balanced picture of what the X800 can do.

 

[Pete]

I agree that this is positive for 3D reviews in general. It'll give consumers a a look from a different angle than most reviews.

 

[THe_KELRaTH]

I've always found that enabling High Detail textures reduces my Radeon 9800 Pro to a crawl under both Unreal and Unreal Tournament.

I use this openGL renderer http://cwdohnal.home.mindspring.com/utglr/
and use the oldskool mod that then runs Unreal (+ expansions) with all the hi res textures too.

http://www.oldunreal.com/patch.shtml

Operation Na Pali Single player game like Unreal (many say it's better)
http://www.planetunreal.com/teamvortex/help/

Single Player game uses oldskool mod (by the guy that did DeCyberxx maps
http://www.ezkeel.com/Design/Legacy/legacy.htm

A new Unreal single player game part 1 finished (of 3)
http://www.fileplanet.com/dl.aspx?/planetunreal/nalichronicles/nalichronicles.zip

Before n after Hi res textures
www.jlmay.f2s.com/UT1-openGLvD3D.jpg

Hi res textures / detail x4 x16 Trueform on (UT Map DM-DeCyberEscher 1998. All the walls of this secret healing room are like moving / rippling water effects).
www.jlmay.f2s.com/UT-Secretroom.jpg

P4-2.4b + 9800Pro

 

[THe_KELRaTH]

I thought 'tween the two of them I got a pretty balanced picture of what the X800 can do.

So we can expect a new site shortly... BeyondOCP

 

[Eronarn]

I thought 'tween the two of them I got a pretty balanced picture of what the X800 can do.

So we can expect a new site shortly... BeyondOCP

Why not just mix some more in?

Tom's BeyondOCPtech 3D!

 

[Reverend]

I'd never advise someone to base a purchase solely off a B3D review

Then you don't think big enough! 8)

On a more serious note, my opinion is basically the same as Dave's - we're talking about folks that prefer to spend more time tinkering with car engines than actually driving it, and those that are the other way round.

It'll be interesting to see the results of HardOCP's new endeavour. I personally would never allow such an approach (benchmark results do not lie if you know exactly what's going on, and you do need to know -- by using synthetic benchmarks -- if a SM 4.0 part is going to do well in SM 4.0 games that you will never be able to buy on the hardware's debut... the important thing is to see if the product hype is justified or not) but then what do I know, eh?

 

[croc_mak]

I really admired Hardocp's work in recent times...They have produced the most useful reviews from a user point of view. The effort taken by Kyle & Co need to be really appreciated by the community IMHO.

I was among many who made a snide remark or two questioning hardocp's tactics last year..but boy did Kyle prove me wrong or what...I have developed great admiration for how hardocp stuck to a theme and I think the results are clearly showing...frankly this is the only site I would recommend today to a friend who asks me for a video card purchase reference.

while I'm a fan of beyond3d and it's reviews too...I have increasingly come to the conclusion that synthetic benchmarks are only useful if you create these tests yourself and understand these tests deeply....Synthetic tests are more like the experiments you do in the chemistry lab to figure out the different elements inside a compound given to you.....

 

[digitalwanderer]

I have increasingly come to the conclusion that synthetic benchmarks are only useful if you create these tests yourself and understand these tests deeply....Synthetic tests are more like the experiments you do in the chemistry lab to figure out the different elements inside a compound given to you.....

"Thud-thud-thud", says the Dig's forehead and desktop in perfect unison.

You're so close, yet still so far.

Sometimes it can be really, really important to know just what those different elements in a compound are and synthetic tests/benchmarks are a very, very good tool for that.

IF used properly, which you are quite correct about. (And you'll note I didn't say a bloody disrespectful or snide thing about [H] either. )

 

[croc_mak]

I have increasingly come to the conclusion that synthetic benchmarks are only useful if you create these tests yourself and understand these tests deeply....Synthetic tests are more like the experiments you do in the chemistry lab to figure out the different elements inside a compound given to you.....

"Thud-thud-thud", says the Dig's forehead and desktop in perfect unison.

You're so close, yet still so far.

Sometimes it can be really, really important to know just what those different elements in a compound are and synthetic tests/benchmarks are a very, very good tool for that.

IF used properly, which you are quite correct about. (And you'll note I didn't say a bloody disrespectful or snide thing about [H] either. )

You are mis-understanding me...I totally agree with you here..Many times it is very important to figure the elements in a compound...My main point(putting it in another way) is that you can get to this goal only by constantly evolving these synthetic tests as new technologies and architectures arise...using the same set of tools over and over again tends towards being less useful is my main point

Also synthetic tests should only be run my people who understand what they are doing....I must admit I'm glad many sites don't do much synthetic tests..there are very few people out there that understand how to do this right..so it's ok that they don't do this..given this situation the sites that just publish canned game demo benchmark results at a few resolutions and declare a winner are less valuable than a site like hardocp is my other main point...

Trust me, the only thing I love more than hardocp right now is some synthetic shader model 3.0 tests run on NV40 hardware and compared with equivalent 2.0 shaders on NV & X800.......

 

[Quitch]

I've always found that enabling High Detail textures reduces my Radeon 9800 Pro to a crawl under both Unreal and Unreal Tournament.

I use this openGL renderer http://cwdohnal.home.mindspring.com/utglr/
and use the oldskool mod that then runs Unreal (+ expansions) with all the hi res textures too.

http://www.oldunreal.com/patch.shtml

Operation Na Pali Single player game like Unreal (many say it's better)
http://www.planetunreal.com/teamvortex/help/

Single Player game uses oldskool mod (by the guy that did DeCyberxx maps
http://www.ezkeel.com/Design/Legacy/legacy.htm

A new Unreal single player game part 1 finished (of 3)
http://www.fileplanet.com/dl.aspx?/planetunreal/nalichronicles/nalichronicles.zip

Before n after Hi res textures
www.jlmay.f2s.com/UT1-openGLvD3D.jpg

Hi res textures / detail x4 x16 Trueform on (UT Map DM-DeCyberEscher 1998. All the walls of this secret healing room are like moving / rippling water effects).
www.jlmay.f2s.com/UT-Secretroom.jpg

P4-2.4b + 9800Pro

Thanks, I'll try and remember all these links

 

[WaltC]

...
while I'm a fan of beyond3d and it's reviews too...I have increasingly come to the conclusion that synthetic benchmarks are only useful if you create these tests yourself and understand these tests deeply....Synthetic tests are more like the experiments you do in the chemistry lab to figure out the different elements inside a compound given to you.....

I can't really agree with this on two levels:

First, the majority of people spending $400-$500US on a 3d card are not going to be in the "casual gamer" class--they are going to want the nuts and bolts before laying out the cash for that kind of product. So I think a mistake that many 3d sites make is in failing to distinguish the different kinds of markets surrounding the different price points for the products they review. Casual reviews seem best suited for casual gamers, imo, of the type who buy in the $200US and < price range.

The more expensive the product, the more the consumer is likely to want to know about the product. I think that's a fairly straightforward proposition, regardless of what products are involved (televisions, stereos, cars, houses, 3d accelerators, cpus, makes no difference, the principle is the same.) In short, a thorough review of the GFFX 5200 or the Radeon 9000 would be underkill when applied to a GFFX5900 or a 9800XT; and a thorough review method for the more expensive products would be overkill for the el cheapos--because not only are the prices different, but the functionalities and capabilities are different, too. So, first of all, one size does not fit all.

Second, take the case of 3dMark last year, as a synthetic benchmark. Long before we saw it proved in games, the benchmark gave us very accurate and reliable information as to the ps2.0 and DX9 full-precision performance of both R300+ and nV30+. Nothing that came afterwards in relation to actual 3d-game support of these features disproved or in any way ever contradicted the initial findings discovered through use of 3dMk03 relative to nV3x and R3x0. I could understand a web site declaring that it wouldn't use 3dMk03, for instance, had real game support of DX9 full precision and ps2.0 indicated the opposite of the synthetic--but such was definitely not the case, was it? I mean, for a site to say that it is categorically dumping "synthetics" sounds very much like 3dMk03--certainly the best *3d-card* benchmark FM has ever done, I think--is being penalized for having been too accurate and having done its job too well. That's why it doesn't make a lot of sense to me to categorically *dump* synthetic benchmarks, when they prove their predictive worth as 3dMK03 certainly proved its own last year. I think software should be dumped if it doesn't work or it produces false results--certainly. But to dump a category of testing software from a review process because it did its job too well? That doesn't compute for me...

I mean, it really, really is impractical to say the least. What's next? Do we "dump" all the TWIMTBP and "Get in the game" games? Do we disqualify Doom3 as "appropriate" testing software because it contains nV40 optimizations--or "dump" HL2 because it contains R4x0 optimizations? Seems like we'd have to, doesn't it? That's what really has me scratching my head over the "dump the synthetics but nothing else" point of view: it's not clear that consumers will benefit from it, and so, if not, then who benefits and why?