Benchmarking with cutscenes -- what's your opinion?

[Brandon]

I hope Dave and Rev don't mind, but I was hoping to get more thoughts on the whole benchmarking with game cutscenes dilemna. I know B3D's and FS' opinion on the matter -- NO!, but I'd like to hear what others think.

In case you didn't know, some hardware websites are beginning to run benchmarks based on game cutscene performance. In some cases it's merely the intro of the game when you load it, in others it may be a demo that you can play back. At first it was just GamePC with NOLF 2 and Freelancer, but now I see that Anandtech is doing the same as well. IMO, this is the wrong way to benchmark, as testing with a cutscene isn't indicative of actual game performance. In fact it isn't anywhere close in a lot of cases. In some ways it's similar to writing a movie review based on a 30-second trailer!

This is also dangerous, as the next thing you know IHVs will spend more time optimizing for game cutscenes. This doesn't benefit anyone who actually plays the game.

So what are your thoughts, am I being too critical? Are the tradeoffs offset by the wider mix of games? Did you even know that a lot of those benchmarks were based on game cutscenes? I'm very curious...

 

[nelg]

I share the same concerns as you. Bad idea! Sounds like an even easier place to cheat.

 

[Joe DeFuria]

Yes, bad idea.

Static clip planes all over again....

 

[Rodéric]

It's a bad, since in-game engine made cutscene have many parameters tweaked (max LOD models...) to make it look good, and there's often little camera move.

So it's absolutely not representative of how the game plays.

Also pre-recorded flypath are a problem due to nVidia's cheats.

 

[Hyp-X]

it's absolutely not representative of how the game plays.

I 2nd that!

 

[WaltC]

'''
...In case you didn't know, some hardware websites are beginning to run benchmarks based on game cutscene performance. In some cases it's merely the intro of the game when you load it, in others it may be a demo that you can play back. At first it was just GamePC with NOLF 2 and Freelancer, but now I see that Anandtech is doing the same as well. IMO, this is the wrong way to benchmark, as testing with a cutscene isn't indicative of actual game performance. In fact it isn't anywhere close in a lot of cases. In some ways it's similar to writing a movie review based on a 30-second trailer!
...
So what are your thoughts, am I being too critical? Are the tradeoffs offset by the wider mix of games? Did you even know that a lot of those benchmarks were based on game cutscenes? I'm very curious...

Thank you much for this info, Brandon. This is the first I've heard about it. No, I don't think you're being too critical. Your opinion is right on the money.

'scuse me, while I take my heart meds....sheesh, man--what next? I'm getting tired, really weary, of saying "Just when I thought I'd seen it all..." It's become apparent to me that I haven't seen anything yet. If there's some way to lower the bar, by golly somebody's going to figure out an innovative way to do it, seems like.

You know, I think I'd like to fire everybody on the Internet with a web site--except you, B3d and maybe one other site somewhere. Yep, that's the ticket...clean house...fire 'em all...with extreme prejudice! It's making me ashamed of being a netizen.

 

[mczak]

It's a bad, since in-game engine made cutscene have many parameters tweaked (max LOD models...) to make it look good, and there's often little camera move.

So it's absolutely not representative of how the game plays.

Are you sure this is true for all games? the nwn cutscenes (which anandtech used) look pretty much the same as the game imho. Still, I agree it isn't exactly representative (the game also doesn't need to calculate AI etc. which would make it more like "flyby" instead of "botmatch" when speaking in UT terms).

Also pre-recorded flypath are a problem due to nVidia's cheats.

That's not worse than using vendor-supplied timedemos though.

Let's hope a reviewer doesn't mistakenly bench an in-game movie ("gee the framerate is a constant 25fps on all graphic cards. Probably cpu limited...")

 

[nelg]

Yes, bad idea.

Static clip planes all over again....

What scares me is that I have a feeling that static clip planes will become a new marketing buzzword. "THE NEW FX5950 WITH CINEFX 2.1 AND SCP 1.1"

 

[Brandon]

Well, it's good to see that B3D and FS aren't alone here, the comments on Anand's seem to be positive so I guess they either have a different opinion or didn't know. As a gamer, I personally think it's the wrong thing to do.

As far as mczak's comments, I agree with your sentiment on NWN, his benching would be the equivalent of running flybys with UT2K3. Not as bad, but not quite reflective of true gameplay either. As far as vendor supplied demos, I figured I'd share this bit of info with you on Halo. As it stands, you can only bench with the demos that came with the game, recording isn't possible. They will be releasing a patch that will address this however, but only for multi. Here's their stance on the status of the current bench:

The benchmark we put in Halo is very complete - every shader type and
rendering trick in the game is represented in the benchmark demo.


If a hardware vendor optimizes a video card so that the benchmark runs
faster, they will be making the whole game run faster by default.

 

[jimbob0i0]

Well, it's good to see that B3D and FS aren't alone here, the comments on Anand's seem to be positive so I guess they either have a different opinion or didn't know. As a gamer, I personally think it's the wrong thing to do.

As far as mczak's comments, I agree with your sentiment on NWN, his benching would be the equivalent of running flybys with UT2K3. Not as bad, but not quite reflective of true gameplay either. As far as vendor supplied demos, I figured I'd share this bit of info with you on Halo. As it stands, you can only bench with the demos that came with the game, recording isn't possible. They will be releasing a patch that will address this however, but only for multi. Here's their stance on the status of the current bench:

The benchmark we put in Halo is very complete - every shader type and
rendering trick in the game is represented in the benchmark demo.


If a hardware vendor optimizes a video card so that the benchmark runs
faster, they will be making the whole game run faster by default.

Unless they know where the camera will be and the new technology of SCP 1.1 can be put into action......

Did the devs miss the last year or something?

 

[CorwinB]

will be releasing a patch that will address this however, but only for multi.

Until the patch is pulled, of course...

If a hardware vendor optimizes a video card so that the benchmark runs
faster, they will be making the whole game run faster by default.

Note that for Halo, putting static clip planes (new updated SCP 1.2 !!!) in all the 3 different rooms would indeed improve the game for everyone. That's a game where butcheatoptimizations will be great.

 

[Simon F]

Yes, bad idea.

Static clip planes all over again....

But Joe, think of the opportunity for the next generation of technology - Dynamic CP.

 

[jb]

Brandon,

I agree thats not the best idea. I also know that some games, Unreal2 for example, were a bit closer to real game in their cut sceen. However what else can you do? With out a time demo, then how do you run a benchmark. Do you go and run around a level every time and hope you press all the right keys at the same time to make it fair? Do you just find a level with a lot of scripted sequences that you can run up and watch (like the first level of AVP2 when you playing the marine)? Just trying to play devil's advocate.

 

[Joe DeFuria]

Yes, bad idea.

Static clip planes all over again....

But Joe, think of the opportunity for the next generation of technology - Dynamic CP.

Hey everyone, look! Simon is dropping hints again on Series5 technology!

 

[Bouncing Zabaglione Bros.]

Brandon,

I agree thats not the best idea. I also know that some games, Unreal2 for example, were a bit closer to real game in their cut sceen. However what else can you do? With out a time demo, then how do you run a benchmark. Do you go and run around a level every time and hope you press all the right keys at the same time to make it fair? Do you just find a level with a lot of scripted sequences that you can run up and watch (like the first level of AVP2 when you playing the marine)? Just trying to play devil's advocate.

Have you ever heard the saying: "If the only tool you have is a hammer, then everything looks like a nail" ? ie, use the right tool for the job at hand.

If you can't get a game-like experience to bench against, then you shouldn't use it for benches. Benches against cutscenes tell us nothing except how a given card runs cut-scenes, where all the settings and loading of the VPU and CPU can be completely different from anything you can experience while playing the game.

To benchmark a cutscene and then claim this proves one card is better at running the game than another is ludicrous. What next, bench the movies spooled off disk and proclaim one card faster? Bench the speed that the menus drop down when changing your screen resolution?
You'd be more accurate running FRAPS and looking at the average FPS after playing the game for half an hour.

This is just an indication of websites scrabbling around trying to find an alternative to the standard benchmarking that Nvidia managed to destroy, and coming up with a solution that shows how woefully ignorant many of these websites are. They are looking for exclusives and trying to bench games that do not have any kind of usable benchmarking mechanism that can tell you any useful information about performance while you play the game.

 

[jimbob0i0]

Yes, bad idea.

Static clip planes all over again....

But Joe, think of the opportunity for the next generation of technology - Dynamic CP.

Well if it was possible to detect exactly what is in the camera and outside it in a scene within a driver and then clip the bits outside (apart from light sources that could effect what is in teh scene) automatically it would be great and would help performance in all situations.... but I cann but help think how horrible it would be to even attempt to implement that - very, very complicated... it would also be heavy on CPU use and that will probably slow things down in the end.

 

[demalion]

My opinion:

Benchmarking cutscenes seems very likely to be useless for representing game performance.

Benchmarking cutscenes seems very capable of being as useful for representing graphics card performance comparisons as its graphical workload, as long as this usage and the relationship to game performance is analyzed and communicated effectively.

...

Any fixed path benchmarking can be distorted by something of the nature of nVidia's static clip plane "innovation". It being a cutscene isn't the reason, it is the fixed path in the cutscene being used as a benchmark, and one particular IHV (so far) deciding they don't want people to see unadjusted results.

 

[Simon F]

Yes, bad idea.

Static clip planes all over again....

But Joe, think of the opportunity for the next generation of technology - Dynamic CP.

Well if it was possible to detect exactly what is in the camera and outside it in a scene within a driver and then clip the bits outside

Well that is exactly what the standard camera fustrum clipping is all about.

(apart from light sources that could effect what is in teh scene)

Actually, light sources are not visible objects and so aren't relevant to this.

 

[nelg]

Have you ever heard the saying: "If the only tool you have is a hammer, then everything looks like a nail" ? ie, use the right tool for the job at hand.

I had never heard that saying before. That explains alot about all the landscaping I did this summer.

 

[Brandon]

Brandon,

I agree thats not the best idea. I also know that some games, Unreal2 for example, were a bit closer to real game in their cut sceen. However what else can you do? With out a time demo, then how do you run a benchmark. Do you go and run around a level every time and hope you press all the right keys at the same time to make it fair? Do you just find a level with a lot of scripted sequences that you can run up and watch (like the first level of AVP2 when you playing the marine)? Just trying to play devil's advocate.

My stance is if you can't accurately reflect gameplay, don't bench with it. I understand that readers want more games and a wider variety of genres in reviews, but I think you do more harm than good if your tests aren't reflective of the true gameplay experience.