Brent....what a fine review you have posted.....

[Mintmaster]

Brent,

I was wondering why you never mention ATI's ability to do the same pseudo-trilinear filtering anymore. Your previous article showed that when you match the quality to NVidia's trilinear, RV350 gets quite a boost in performance.

I haven't seen what effect this has on the 9600XT yet, and it would have been great to see numbers directly comparable to the 5700 in this review. Other than that, great review. I also like how you don't exagerrate small framerate differences to be a "win" for one card over the other.

 

[Fodder]

One thing I've noticed in [H] reviews and don't think much of is benching games that don't have demo capabilities. Sure you can run through the track/level and get a rough idea of how it runs, but if it doesn't render the exact same things, how are the results comparable?

At least, I think that's what you're doing when you mention a 'manual runthrough'.

 

[digitalwanderer]

One thing I've noticed in [H] reviews and don't think much of is benching games that don't have demo capabilities. Sure you can run through the track/level and get a rough idea of how it runs, but if it doesn't render the exact same things, how are the results comparable?

At least, I think that's what you're doing when you mention a 'manual runthrough'.

Funny, that's the part I really LIKE about Brent's review!

Yes a demo will give you the exact same rendered scene, but that will also allow IHVs to optimize for that exact same scene. Using fraps on a manual run-through (and by "manual run-through" I mean by sitting down and playing a game with fraps running) is an excellent way to measure performance.

Yes, there are ways to abuse fraps to misrepresent the results; yes, the run-throughs aren't exactly the same (but in my own testing I've found them to be damned close if I run the same section the same way); and yes, the results even if everything is done right can be misleading if you choose the wrong section of the game to bench...but frapsing the way Brent does is an excellent way to gauge/show performance if the reviewer is careful to keep it fair and accurate. (I'm benching games using fraps for my upcoming review, but I'm not doing the second-by-second breakdown that Brent does...I prefer the min/max/avg numbers from a brief run.)

I really do like fraps as a way to benchmark, but without recordable demos to bench there is a bit of a fine art to making fraps accurately represent gameplay.

 

[Tim Murray]

Yes a demo will give you the exact same rendered scene, but that will also allow IHVs to optimize for that exact same scene. Using fraps on a manual run-through (and by "manual run-through" I mean by sitting down and playing a game with fraps running) is an excellent way to measure performance.

psst, if it's a recorded demo, then an IHV "optimizing" for that demo will improve actual gameplay performance (unless they use clip planes, which would be more than a little odd).

but anyway, FRAPS manual runs work because of the law of averages--if you do almost the same things every time, it evens out in the average. really, it does work--you can try it yourself.

 

[digitalwanderer]

Yes a demo will give you the exact same rendered scene, but that will also allow IHVs to optimize for that exact same scene. Using fraps on a manual run-through (and by "manual run-through" I mean by sitting down and playing a game with fraps running) is an excellent way to measure performance.

psst, if it's a recorded demo, then an IHV "optimizing" for that demo will improve actual gameplay performance (unless they use clip planes, which would be more than a little odd).

Ah, but hasn't one of them pesky IHV's already proved that they can be "a little odd" when it comes to creative driver optimizations?

Besides, the IHV optimizing for the game will be reflected by the FRAPS method I outlined too. (And I agree about the law of averages making it all work out, the hard part for me is playing the game with fraps on long enough to find a spot in the game to do the fraps run that I think fairly represents the gaming experience...but it's kind of fun too. )

 

[Fodder]

Yes a demo will give you the exact same rendered scene, but that will also allow IHVs to optimize for that exact same scene.

Simple, use your own demo. I can see where you guys are coming from though, I suppose it's up to personal taste and whether you're a perfectionist bastard like me.

 

[Tim Murray]

Diggy, I don't think that a driver could detect a specific recorded demo for a game--there would have to be... individual frame detection? something crazy like that. plus, it'd be REALLY easy to discover if they were using clip planes.

 

[Bouncing Zabaglione Bros.]

Diggy, I don't think that a driver could detect a specific recorded demo for a game--there would have to be... individual frame detection? something crazy like that. plus, it'd be REALLY easy to discover if they were using clip planes.

All they'd have to do is detect FRAPS running. IIRC, it's not like Nvidia hasn't already made noises about FRAPS and benchmarking

 

[Tim Murray]

Diggy, I don't think that a driver could detect a specific recorded demo for a game--there would have to be... individual frame detection? something crazy like that. plus, it'd be REALLY easy to discover if they were using clip planes.

All they'd have to do is detect FRAPS running. IIRC, it's not like Nvidia hasn't already made noises about FRAPS and benchmarking

gwah? how is detecting FRAPS going to let an IHV detect a specific demo for a specific game and then apply clip planes of optimizations specific to that demo?

 

[digitalwanderer]

gwah? how is detecting FRAPS going to let an IHV detect a specific demo for a specific game and then apply clip planes of optimizations specific to that demo?

I wouldn't be too shocked if an IHV found a way to detect FRAPS and was able to figure out a way to have their drivers tweak the register fraps uses to count frames to make their score turn out however much higher they wanted.

Sure it would be possible to find out about it, but so was finding out about 3dm2k3 cheats and we all know who won that round.

What I find sad about it all is the effort and time wasted just to do that when it could be better spent improving actual driver performance.

 

[digitalwanderer]

Yes a demo will give you the exact same rendered scene, but that will also allow IHVs to optimize for that exact same scene.

Simple, use your own demo. I can see where you guys are coming from though, I suppose it's up to personal taste and whether you're a perfectionist bastard like me.

No argument from me on that, if a game has the option to record your own demo than that is DEFINATELY the way to go and any reviewer who doesn't run a custom bench is an idiot. (Or using a "standard" demo just to give some numbers for cross-comparing purposes, but they should include a custom run-thru than too.)

The big problem is more and more games are shipping without that feature and I hate just limiting myself to games that have built in benchmarking/demo recording features since most of the games I actually play ain't got them...in my review I'm aiming to show what a card can do in a couple of three different systems across a spectrum of games I currently play and I decided it would be better to find a way to benchmark any game I wanted than to restrict myself to titles that made it easy. (And I'm only about half a year behind on the bloody thing too, but it's almost done. )

 

[Tim Murray]

gwah? how is detecting FRAPS going to let an IHV detect a specific demo for a specific game and then apply clip planes of optimizations specific to that demo?

I wouldn't be too shocked if an IHV found a way to detect FRAPS and was able to figure out a way to have their drivers tweak the register fraps uses to count frames to make their score turn out however much higher they wanted.

Sure it would be possible to find out about it, but so was finding out about 3dm2k3 cheats and we all know who won that round.

What I find sad about it all is the effort and time wasted just to do that when it could be better spent improving actual driver performance.

oh, monkeying with the frame count? sure, it'd be easy. but think about it--run FRAPS and a UT2004 benchmark at the same time and see if they line up. if they do, dandy. if they don't, well then, you have a problem somewhere.

 

[Brent]

Jeez, Brent, the 5700 on FS2004 absolutely sucks...... Where's that famous much better ansotropic filtering......., or am I missing something here?

Your not missing anything, it really is that bad. And i've found out it is bad in some other games too which I'll discuss with screenschots in an upcoming review.

Brent,

I was wondering why you never mention ATI's ability to do the same pseudo-trilinear filtering anymore. Your previous article showed that when you match the quality to NVidia's trilinear, RV350 gets quite a boost in performance.

I haven't seen what effect this has on the 9600XT yet, and it would have been great to see numbers directly comparable to the 5700 in this review. Other than that, great review. I also like how you don't exagerrate small framerate differences to be a "win" for one card over the other.

We are testing real world gaming performance and set the cards up as the end user would use them. The default settings on the ATI cards are the texture sliders at their maximum values. The only thing we change is to disable VSYNC. Basically each cards control panel is at their highest settings allowable so we can compare performance and IQ at the highest levels they allow.

Not many people are gonna grab their new shiny 9600XT and then set the textures to a lower quality to play games. They want to know how the card performs at the highest settings, best image quality the card can produce, and then compare the performance and IQ.

I'm a firm believer in high image quality and always wanting more better graphics. I won't be satisified until we have holodeck type realness

One thing I've noticed in [H] reviews and don't think much of is benching games that don't have demo capabilities. Sure you can run through the track/level and get a rough idea of how it runs, but if it doesn't render the exact same things, how are the results comparable?

At least, I think that's what you're doing when you mention a 'manual runthrough'.

In the case that we can't make a recorded demo. We make sure to start in exactly the same place each time, we run through the same path and end at the same place each time. We do this at least 3 times.

One thing I've noticed in [H] reviews and don't think much of is benching games that don't have demo capabilities. Sure you can run through the track/level and get a rough idea of how it runs, but if it doesn't render the exact same things, how are the results comparable?

At least, I think that's what you're doing when you mention a 'manual runthrough'.

Funny, that's the part I really LIKE about Brent's review!

Yes a demo will give you the exact same rendered scene, but that will also allow IHVs to optimize for that exact same scene. Using fraps on a manual run-through (and by "manual run-through" I mean by sitting down and playing a game with fraps running) is an excellent way to measure performance.

Yes, there are ways to abuse fraps to misrepresent the results; yes, the run-throughs aren't exactly the same (but in my own testing I've found them to be damned close if I run the same section the same way); and yes, the results even if everything is done right can be misleading if you choose the wrong section of the game to bench...but frapsing the way Brent does is an excellent way to gauge/show performance if the reviewer is careful to keep it fair and accurate. (I'm benching games using fraps for my upcoming review, but I'm not doing the second-by-second breakdown that Brent does...I prefer the min/max/avg numbers from a brief run.)

I really do like fraps as a way to benchmark, but without recordable demos to bench there is a bit of a fine art to making fraps accurately represent gameplay.

The demos that we do make, are private. In fact I'm the only one that has them.

If I could encourage developers to include one thing in their games it would be to include a way to make recorded demos in that game and to make save game places at will so screenshots can be taken from the exact same place each time.

 

[Bouncing Zabaglione Bros.]

gwah? how is detecting FRAPS going to let an IHV detect a specific demo for a specific game and then apply clip planes of optimizations specific to that demo?

Not clip planes specifically, but you could assume that the use of FRAPS means the user is doing some kind of benchmark, and activate various cheats. We've seen it happen where Nvidia detects benchmarks modes in games, fiddles with the IQ to get higher scores, and then gets lower performance in-game because those cheats are no longer active.

 

[Tim Murray]

gwah? how is detecting FRAPS going to let an IHV detect a specific demo for a specific game and then apply clip planes of optimizations specific to that demo?

Not clip planes specifically, but you could assume that the use of FRAPS means the user is doing some kind of benchmark, and activate various cheats. We've seen it happen where Nvidia detects benchmarks modes in games, fiddles with the IQ to get higher scores, and then gets lower performance in-game because those cheats are no longer active.

but like I said, that'd be easy to detect. and if it reduced IQ, well, you could record a movie from FRAPS and detect it pretty easily.

it seems like you're preparing for a somewhat bizarre development that would be incredibly easy to find (no special versions of anything needed--if you're going from a recorded demo, you could take screenshots with FRAPS running and without FRAPS running and compare the two. simple as that.). it makes no sense to me. maybe some goon from one of the bigger sites who has someone else run all the benchmarks for him and doesn't pay any attention whatsoever would fall for it, but I don't think any knowledgeable reviewer would.

NVIDIA reviewers' guides also recommend the use of FRAPS with some games that don't have a built-in benchmark but do have demo recording capabilities.

 

[digitalwanderer]

it seems like you're preparing for a somewhat bizarre development that would be incredibly easy to find (no special versions of anything needed--if you're going from a recorded demo, you could take screenshots with FRAPS running and without FRAPS running and compare the two. simple as that.). it makes no sense to me. maybe some goon from one of the bigger sites who has someone else run all the benchmarks for him and doesn't pay any attention whatsoever would fall for it, but I don't think any knowledgeable reviewer would.

Baron, do you remember the last year and a half? :|

The graphic world is goinog thru some rather bizarre times right now, expecting the unexpected seems like the best way to stay on top of things. 8)

 

[Tim Murray]

Baron, do you remember the last year and a half? :|

The graphic world is goinog thru some rather bizarre times right now, expecting the unexpected seems like the best way to stay on top of things. 8)

I remember the last year and a half. I'm saying that the NVIDIA specifically makes optimizations hard to detect (not filename-based, but window-title-based and then encrypted so it can't be detected by running strings or something like that). A FRAPS cheat would be so ridiculously easy to detect, and with the force trilinear option appearing in leaked drivers (hey Stealthy, remember how I said that was going to happen two months before the NV40 came out? FUNKY.), it seems that NV is trying to regain credibility with high-end users before the NV40.

Nevertheless, a FRAPS cheat is *possible*, I'll grant you that. But if there is one, it'll be discovered the instant someone uses FRAPS with that driver version.

 

[Bouncing Zabaglione Bros.]

but like I said, that'd be easy to detect. and if it reduced IQ, well, you could record a movie from FRAPS and detect it pretty easily.

it seems like you're preparing for a somewhat bizarre development that would be incredibly easy to find (no special versions of anything needed--if you're going from a recorded demo, you could take screenshots with FRAPS running and without FRAPS running and compare the two. simple as that.). it makes no sense to me.

Look at what happened around the Nvidia Brilinear issue. Even when it was obvious that trilinear requests from the app were being bastardised, we still had some reviewers from well known sites claiming that they couldn't see the difference. IIRC, the brilinear issue first came up in benchmark mode only. Nvidia then "legitimised" this behaviour by making it the default for UT2K3, and then all Direct X apps.

Is that not bizarre enough for you?

 

[digitalwanderer]

but like I said, that'd be easy to detect. and if it reduced IQ, well, you could record a movie from FRAPS and detect it pretty easily.

it seems like you're preparing for a somewhat bizarre development that would be incredibly easy to find (no special versions of anything needed--if you're going from a recorded demo, you could take screenshots with FRAPS running and without FRAPS running and compare the two. simple as that.). it makes no sense to me.

Look at what happened around the Nvidia Brilinear issue. Even when it was obvious that trilinear requests from the app were being bastardised, we still had some reviewers from well known sites claiming that they couldn't see the difference. IIRC, the brilinear issue first came up in benchmark mode only. Nvidia then "legitimised" this behaviour by making it the default for UT2K3, and then all Direct X apps.

Is that not bizarre enough for you?

It's pretty bizarre, but I still got this weird hunch that nVidia is going to top it in the next round.

 

[Tim Murray]

to the best of my knowledge brilinear started in UT2003, both in the application and in the benchmark executable.

but no preventative measures can take the place of an intelligent reviewer.