A new modern engine benchmark

[Doomtrooper]

Comanche 4 benchmark utility, something OTHER than Quake3 :)

http://www2.novalogic.com/downloads.html#c4demo

Radeon 8500 users make sure you edit the C4.cfg, Although C4 detects shaders it disables them since this was designed on a Geforce 3.
You have to go into C4.cfg and set Disable_Shaders=1 to 0

[Dave Baumann]

Fortuitous timing - I was just wondering what other benchmarks I could use for this Radeon review. Even more useful to see what the Pixel Shader hit is like. :)

[Dave Baumann]

Thank christ for that as well - it has a batch mode as well! :D

[kl899]

is there an alternative download site? gigex isn't working :(

[Doomtrooper]

Demo looks sweet but I think the engine needs some platform optimization work. I get 30 fps @ 1024 and 37 fps @ 800 x 600. I also don't like having to edit the .cfg file to enable shaders, whats the game look for when checking the card, DX compatability or just looks for a Geforce 3.

Anyhow nice to see a new benchmark anyways.

[rhink]

grr.... is there anywhere to download it without using that gigex crap?

[Doomtrooper]

None so far, Gigex seems to work ok for me. 15 second download and away you go ?? :-?

[rhink]

it ain't working for me... won't connect once the pos gigex xfer utility downloads.

[msan_ewu]

I hate Gigex too but downloaded it anyway.... I had a 24.27fps score but I'm sure most will post higher as I didn't drop any IQ settings. It sure felt faster than 25fps though... smooth

The demo is nice but nothing special compared to what's out already. I liked the water and the rotor wash though :D

Tbird@1.2
GF3
512 pc-133
XP

edit: my settings were all default i.e. 8x6, etc....

[Randell]

its strange, I get ~28 without AA or aniso, ~25 with 2xQualityAA and 16x aniso.

nvNews forums seem to be full of people with Gf3/8500 class hardware getting 25-30 fps, but a slew of Ti200 owners getting 37+fps!

[Ante P]

There's some guy over at nvnews with a 2.2 Ghz P4 and a overclocked Ti 4600 and he just managed to pass the 40 fps mark

something is not right
but if you look at the scene shifting you'll see that it actually blends two fully rendered scenes
crazy stuff that'll never happe during actual gameplay

it also seems like it's very CPU limited

[Mephisto]

What a cool benchmark! That's what we need, a modern dx8 benchmark made directly from a scene within a game! Finally, good bye Quake 3!

[Tahir2]

This benchmark is a load of crap. The game is meant to be fun to play but it doesn't look too hot IMO. I don't think the results are really all that useful. The only thing it tells me is that the graphics engine needs some serious attention. Dave I don't reccommend using it in your review.
Doesn't tell you anything useful really.


Results below:

33 fps at 800*600
30 fps at 1024*768

CPU - AthlonXP @ 1564.54 MHz (10.5*148.91)
Motherboard - EPoX 8KHA+ KT266A
RAM - Nanya 512MB PC2100 at CAS 2 148.91 MHz DDR
RadeonLE GPU at 295MHz RAM at 270MHz

Using the 6025 XP driver for the Radeon (I like to play DVD with hardware enabled and later drivers seem to have screwed that up).

Regards,

[Entropy]

This benchmark is a load of crap. The game is meant to be fun to play but it doesn't look too hot IMO. I don't think the results are really all that useful. The only thing it tells me is that the graphics engine needs some serious attention. Dave I don't reccommend using it in your review.
Doesn't tell you anything useful really.

I'd like to echo the above sentiment.
Besides managing to look pretty ugly in spite of an average of 200 000 polygons per frame, it seems to be mostly limited by the CPU/memory/buses(?) subsystem rather than the graphics.

In that respect it is similar to Unreal/UT, only that UT manages to be mostly CPU/memory limited in spite of having a tenth to a hundredth as many polygons per frame.... And still has a hard time achieving consistently good framerates on any hardware in existance.

I guess Carmack is a genious after all, or perhaps simply a competent craftsman among hackers and wanna-bes.

It scares me a bit when Tim Sweeney goes on and on about how many polygons Unreal 2 will use. Will it be another game(engine) that will never run particularly well on any hardware in relation to how it looks? I remember when Tim first started talking about the new engine and said that it wouldn't use any method of reducing the number of polygons for distant terrain because "it never looked good" but instead they would rely on the GPUs to handle it fast enough. Amazing to hear a man _brag_ about how they will cripple either performance or visual quality out of laziness.

Oh. I'm ranting. Sorry. Comanche4? Not impressed.

Entropy

[Mephisto]

How can you judge this benchmark based on postings based on different systems? Who knows, one might have AGP1x without SBA and SDR system memory, the other AGP4x with SBA with nForce and DDR. And obviosly the benchmark does different things with different gfx class ( DX7/8 ) hardware, so you would have to set the things manualy in the config file to get really comparable results.

BTW: I just get the impression that some people just don't like the benchmark because they see that their system with silly expensive shader graphics card does not get more thant 30 fps.

[Humus]

I wasn't especially impressed either, neither by graphics or by gameplay. After 5 minutes or so I decided I had enough and uninstalled it.


Entropy, don't forget that the Unreal engine was initially developed mainly for software rendering, then later upgraded to Glide, and then to DirectX/OpenGL later on. The techniques it was based on is much more suitable for software rendering than what Carmacks Q2/Q3 engines were. Unreal had way better graphics that Q2 for instance, with high performance on Voodoo2 and lower hardware. What made Unreal so fast on old cards like V2 was that it has almost no overdraw. But as GPU's continued to get faster very quickly the optimizations in the Unreal engine became a brake instead. Today the GPU sits idle waiting for the CPU to sort out visibility in Unreal engine based games that the GPU itself would sort out faster. The quake 2/3 engines put more work on the CPU instead, thus they scale very well with newer GPU's. On my old G400 UT would run much faster than Q3 for instance, on my old Radeon the performance was about the same, on my current Radeon8500 Q3 run much faster.

I don't think we should complain about Tim Sweeney's work, his engine lasted quite long, but it's gotten outdated now. But he has realized that himself too, thus the Unreal2 engine is taking something similar to Carmacks approach. That's probably why he choose not to do any fancy LOD stuff on the terrain, it would probably end up being slower anyway if he spent to much cycles on sorting out how much detail he should draw the terrain with, he's probably only doing some really rough visibility stuff and throws the rest on the GPU, it'll probably be a better solution in the long run.

[Randell]

Well I got bored very quickly with the gameplay in the demo.

People playing the game get much higher fps than they get in the benchmark, utilising the same settings, what.

As for SBA making a differnce between systems? How much?

The reality is across DDR and SDR systems, 1gig Athlons to 2 gig pentiums, almost everybody is throwing out similar results. The one anomaly seems to be Ti200 owners who are claiming higher frame rates than Gf3 and Ti500 owners. Again without side by side comparisons of system specs you 'cant' draw conclusions. Other than some Ti200 owners on nvNews like to brag ;)

[Doomtrooper]

I'm sure we'll see a patch to address some of the issues, this is one of the 1st bechmarks besides Aquamark that uses some advanced effects and I'm surprised how quickly people are writing it off as crapola.

Besides I'm a sucker for seeing things blow up and if I see another Quake 3 timedemo I'm gonna lose it :P

[Entropy]

I don't think we should complain about Tim Sweeney's work, ...

I disagree. :)
Though I can certainly forgive the man that he zigged instead of zagged on a particular technical point, I don't much care for his cathegorical statements, judgemental attitude and recently his infantile harping on how Unreal will have 100 000+ polys per frame.
First off, it's no guarantee that it will look particularly good. The Comanche4 Demo makes that embarrassingly obvious. Secondly, it implies that they don't do much (or any) LOD work, making for a gameworld bogged down with unnecessary detail at high distances in order to look reasonably good close up. Your comments about LOD are not without validity of course, but it is not as if it takes a huge amount of work to do, nor does it lhave to look bad, as long as you are not constrained by trying to reduce the scene to an absolute bare minimum number of polygons. Safe, conservative LOD is way better than none at all. Not using it is asking for trouble when you are creating a general engine, or an engine intended for multiplayer use.

Have _anyone_ here seen good framerates in any games when the polycounts climb beyond 6 digits? No, benchmarks such as 3DMarks don't count. Not too many such games around of course, and those that are may not be optimally coded, but.... I think a certain sceptisism is warranted. As long as you can reduce the polycount to improve framerate this won't be a problem, but that reduces all this "over a hundred thousand polygons" to marketing and little else.

We could probably have a good discussion here about how a large polygon budget should be used though. And what bottlenecks/pitfalls there are. New thread, someone?

And I'm sorry, but I can't help but comparing Sweeneys posturing with Carmacks nerdy :) description of how they make relatively few polys look really good, and how they allocate the available computing budget. it's probably a bit silly of me, but I know which attitude I'm comfortable with.

Entropy

[Dave Baumann]

BTW: I just get the impression that some people just don't like the benchmark because they see that their system with silly expensive shader graphics card does not get more thant 30 fps.

No, the benchmark is clearly CPU limited – mapping out the fillrate graph shows some real oddities with the Radeon 8500 because the drop off at 1600x1200 is far too much. Also the Pixel Shader don’t make much sense as so far all the testing indicates that its actually faster with PS enabled than disabled!

[Oompa Loompa]

It scares me a bit when Tim Sweeney goes on and on about how many polygons Unreal 2 will use. Will it be another game(engine) that will never run particularly well on any hardware in relation to how it looks?

Hmm. Well, Unreal/UT may have never been a good benchmark, but these were good games, and licensees made at least as many good looking and performing UT-engine games as Q3-engine games. So I'm not sure all of your criticism is justified.

But in any case, things seem to be different this time. According to the "Unreal Peformance Test 2002", results seem to scale very much according to GPU power. http://www.anandtech.com/video/showdoc.html?i=1583&p=12

(Hmm. When did Anand start embedding his charts in such a sneaky fashion? As far as I can see the charts don't exist as discrete GIFs or JPEGs, but each bar is a separate object. Makes it hard to steal :-))

[darkblu]

We could probably have a good discussion here about how a large polygon budget should be used though. And what bottlenecks/pitfalls there are. New thread, someone?

that'd be interesting.

[Mephisto]

No, the benchmark is clearly CPU limited – mapping out the fillrate graph shows some real oddities with the Radeon 8500 because the drop off at 1600x1200 is far too much. Also the Pixel Shader don’t make much sense as so far all the testing indicates that its actually faster with PS enabled than disabled!

Sure? here with my GF3 on a Thunderbird 1 GHz (<- !) it definitly shows a fillrate limited fall down.

May be you shoud enable the vertex shading on R8500 (off by default ...).

[Reverend]

The benchmark itself isn't indicative of gameplay. I have the full retail version and the gameplay really is rather good, being a good mix of straight-out action that many FPS'ers will be familiar with and that of a "sim", a not-so-serious one at that however. That's my opinion of gameplay and as usual gameplay is more subjective than something like IQ.

That said, the IQ is, well, simple. It is nothing to shout about. The only stuff worth mentioning are the rotor water vertex effects (which aren't that realistic anyway) and highish polies. Everything else seem dated to me (look at the explosion effect of the ship when it eventually goes down during the benchmark... it's cartoonish).

The full retail version is like this demo/benchmark when it comes to performance - it is incredibly CPU and bus limited and the explosion effects gobbles up fillrate (which is the primary reason for all the lowest recorded framerates during the benchmark run). I ran a batched benchmark (FYI. I never was able to use the LAUNCH.EXE for batched benchmark so I tried without the LAUNCH.EXE and simply used c4demo.exe for batched benchmarking, which ran fine) :

===========================================

Comanche 4 Benchmark Results for 640x480x32 FSAA=0 VSync=OFF DXTC=ENABLED AUDIO=OFF SHADERS=ENABLED

Frames per second: 40.41 avg
Tris per second: 8,047,552 avg


Comanche 4 Benchmark Results for 1024x768x32 FSAA=0 VSync=OFF DXTC=ENABLED AUDIO=OFF SHADERS=ENABLED

Frames per second: 40.30 avg
Tris per second: 8,025,587 avg


Comanche 4 Benchmark Results for 1600x1200x32 FSAA=0 VSync=OFF DXTC=ENABLED AUDIO=OFF SHADERS=ENABLED

Frames per second: 37.89 avg
Tris per second: 7,546,306 avg


Comanche 4 Benchmark Results for 640x480x32 FSAA=2 VSync=OFF DXTC=ENABLED AUDIO=OFF SHADERS=ENABLED

Frames per second: 40.17 avg
Tris per second: 8,000,690 avg


Comanche 4 Benchmark Results for 1024x768x32 FSAA=2 VSync=OFF DXTC=ENABLED AUDIO=OFF SHADERS=ENABLED

Frames per second: 39.49 avg
Tris per second: 7,863,706 avg


Comanche 4 Benchmark Results for 1600x1200x32 FSAA=2 VSync=OFF DXTC=ENABLED AUDIO=OFF SHADERS=ENABLED

Frames per second: 29.03 avg
Tris per second: 5,782,285 avg


Comanche 4 Benchmark Results for 640x480x32 FSAA=4 VSync=OFF DXTC=ENABLED AUDIO=OFF SHADERS=ENABLED

Frames per second: 39.91 avg
Tris per second: 7,947,273 avg


Comanche 4 Benchmark Results for 1024x768x32 FSAA=4 VSync=OFF DXTC=ENABLED AUDIO=OFF SHADERS=ENABLED

Frames per second: 35.65 avg
Tris per second: 7,098,793 avg


Comanche 4 Benchmark Results for 1600x1200x32 FSAA=4 VSync=OFF DXTC=ENABLED AUDIO=OFF SHADERS=ENABLED

Frames per second: 16.76 avg
Tris per second: 3,338,692 avg


Comanche 4 Benchmark Results for 640x480x32 FSAA=0 VSync=OFF DXTC=ENABLED AUDIO=OFF SHADERS=DISABLED

Frames per second: 38.76 avg
Tris per second: 7,676,165 avg


Comanche 4 Benchmark Results for 1024x768x32 FSAA=0 VSync=OFF DXTC=ENABLED AUDIO=OFF SHADERS=DISABLED

Frames per second: 38.65 avg
Tris per second: 7,655,600 avg


Comanche 4 Benchmark Results for 1600x1200x32 FSAA=0 VSync=OFF DXTC=ENABLED AUDIO=OFF SHADERS=DISABLED

Frames per second: 36.89 avg
Tris per second: 7,307,617 avg


Comanche 4 Benchmark Results for 640x480x32 FSAA=2 VSync=OFF DXTC=ENABLED AUDIO=OFF SHADERS=DISABLED

Frames per second: 38.53 avg
Tris per second: 7,630,637 avg


Comanche 4 Benchmark Results for 1024x768x32 FSAA=2 VSync=OFF DXTC=ENABLED AUDIO=OFF SHADERS=DISABLED

Frames per second: 38.04 avg
Tris per second: 7,534,190 avg


Comanche 4 Benchmark Results for 1600x1200x32 FSAA=2 VSync=OFF DXTC=ENABLED AUDIO=OFF SHADERS=DISABLED

Frames per second: 29.10 avg
Tris per second: 5,764,551 avg


Comanche 4 Benchmark Results for 640x480x32 FSAA=4 VSync=OFF DXTC=ENABLED AUDIO=OFF SHADERS=DISABLED

Frames per second: 38.43 avg
Tris per second: 7,612,581 avg


Comanche 4 Benchmark Results for 1024x768x32 FSAA=4 VSync=OFF DXTC=ENABLED AUDIO=OFF SHADERS=DISABLED

Frames per second: 34.77 avg
Tris per second: 6,885,814 avg


Comanche 4 Benchmark Results for 1600x1200x32 FSAA=4 VSync=OFF DXTC=ENABLED AUDIO=OFF SHADERS=DISABLED

Frames per second: 16.93 avg
Tris per second: 3,353,467 avg

===========================================

That's on a XP2000+, 256MB PC2100, GF4 Ti4600 with official 28.32 drivers, DX8.1, WinXP. Sound was disabled via the game's specified command line for doing so and all AA was implemented via command line option rather than thru NV's display properties. That SHADERS= option is a result of using either "dx7" or "dx8" in the command line. It corresponds to dx7=SHADERS=DISABLED and dx8=SHADERS=ENABLED.

A few notes :
- Note the DX7 vs DX8 scores for all tests, AA or NoAA
- WRT the above, I don't believe the game uses any PS effects, only VS. Accordingly, if I'm correct, DX7 would mean vertex shaders using host CPU and DX8 would mean using vid card supporting VS. I also believe that the VS effects largely has to do with the rotor-on-water effects - and using the command line option to use "DX7" or "DX8" resulted in the same such effect. Note Wavey's comments on this - I think it comes down to which is faster for the VS effects used in this particular game/benchmark - the host CPU or the GPU.
- Note triangle throughput differences at diff rez

This is another game that can be indicative of the way the latest crop of games are being developed - they will become more dependant on the host CPU up until a rez like 1024x768. Stuff like physics, sound and AI will ensure CPU limitation. Look at this particular game, MOHAA, Max Payne...

[Humus]

The reality is across DDR and SDR systems, 1gig Athlons to 2 gig pentiums, almost everybody is throwing out similar results.

I suppose that means it's limited by AGP bandwidth. They are probably storing all the geometry in AGP memory, and with that massive polygoncount I suppose it will easily become the bottleneck.

[Humus]

I don't think we should complain about Tim Sweeney's work, ...

I disagree. :)
Though I can certainly forgive the man that he zigged instead of zagged on a particular technical point, I don't much care for his cathegorical statements, judgemental attitude and recently his infantile harping on how Unreal will have 100 000+ polys per frame.
First off, it's no guarantee that it will look particularly good. The Comanche4 Demo makes that embarrassingly obvious. Secondly, it implies that they don't do much (or any) LOD work, making for a gameworld bogged down with unnecessary detail at high distances in order to look reasonably good close up. Your comments about LOD are not without validity of course, but it is not as if it takes a huge amount of work to do, nor does it lhave to look bad, as long as you are not constrained by trying to reduce the scene to an absolute bare minimum number of polygons. Safe, conservative LOD is way better than none at all. Not using it is asking for trouble when you are creating a general engine, or an engine intended for multiplayer use.

Have _anyone_ here seen good framerates in any games when the polycounts climb beyond 6 digits? No, benchmarks such as 3DMarks don't count. Not too many such games around of course, and those that are may not be optimally coded, but.... I think a certain sceptisism is warranted. As long as you can reduce the polycount to improve framerate this won't be a problem, but that reduces all this "over a hundred thousand polygons" to marketing and little else.

We could probably have a good discussion here about how a large polygon budget should be used though. And what bottlenecks/pitfalls there are. New thread, someone?

And I'm sorry, but I can't help but comparing Sweeneys posturing with Carmacks nerdy :) description of how they make relatively few polys look really good, and how they allocate the available computing budget. it's probably a bit silly of me, but I know which attitude I'm comfortable with.

Entropy

I can agree than Tim's attitude isn't the best, I have a lot more respect for Carmack with regards to that. But he have done some really good work, as have Carmack. I'm very excited about both Unreal2/UT2 and Doom3, but I think Doom3 will look much better.

[Randell]

Yes the developers posted that it uses a lot of polygons and is limited by the AGP bus in the thread over at nvNews.

[Dave Baumann]

Sorry to for the OT gripe in the thread, but....

Could everyone please be a little more circumspect in their Quoting - I know its easy just to hit the quote button and reply, but when you do so please can you trim the quotes to the relavent parts.

You guys are eating database space up at a rapid rate! :-?

[Doomtrooper]

Somebody PLEASE leak the UT Technology benchmark, we need a new freakin playtoy and not another review of Quake 3 running at 250 fps..please.... :cry:

[Entropy]

I can agree than Tim's attitude isn't the best, I have a lot more respect for Carmack with regards to that. But he have done some really good work, as have Carmack. I'm very excited about both Unreal2/UT2 and Doom3, but I think Doom3 will look much better.

I apologize for posting while in a grumpy mood, but when doing my gfx-sweep of the web, I first read this interview with Tim Sweeney [http://www.gamespy.com/gdc2002/sweeney/], and then downloaded and ran the Comanche4 demo.

The Comanche4 demo is yet another example of designers using additional graphics performance to do more-of-the-same. We get 100 highly detailed trees in the background rather than 15 cruder ones. Madonion realized that this was a problem with their 3dMark2000, they added their detail in ways that either wasn't percieved at all (i.e. the barrels being smoother) or didn't contribute to scene realism (150 crates and barrels does not make a more realistic dock scene than 20). Their Nature scene in 3DMark2001 on the other hand is an excellent example of gfx power used in an attempt to increase realism in new ways.

Using tons of polygons on static environments solves few actual problems, and bring the potential to create new ones. It is reasonable to assume that at some point in the future, increases in polygon count will level off since it seems wasteful to render multiple polygons per pixel. Our output medium has limited resolution, and textures and bumpmaps can often convey detail information in a way that is convincing. Increases in realism will come from other areas, such as lightning, which has the advantage of being easily expressible in mathematical models even though they may be difficult to implement in an efficient manner on current hardware. More challenging is movement, particularly of characters. Characters and creatures are probably the most difficult, but trees/vehicles/weather/grass/water/cloth et cetera are a bit easier to model, and even crude models are better than nothing. When it comes to characters however I predict that we will soon see 5000-10000 polygon characters that move no more convincingly than 250 polygons of Lara Croft did in the dawn of consumer realtime 3D. And some will probably do worse. This is an example of a problem where we our models are too crude, regardless of how much computing power we throw at the problem. Final Fantasy, the movie, is probably as good as I've seen yet, but their methods may not be easily applicable to real time rendering.

What will definitely not bring us forward one whit is when icons of consumer 3D programming avoid handling terrain LOD and then brag about how their polygon counts have gone up.... going from that interview directly to the Comanche4 demo with its average of 200 000 polygons per frame was a downright depressing example of where such attitudes can take us. Not Very Far At All.

Entropy

[Dave Baumann]

OK, I have a responce as to the performance difference between 'DX&' and 'DX8' numbers, I'll post the full responce in the upcoming Radeon 8500 review.

[jb]

which will be ready for us to look at when good sir?

[Dave Baumann]

With a little luck, later this week/early next week.

[Johnny Rotten]

I think you're perhaps being a little harsh towards Mr. Sweeney, or at least interpreting his statements as being more bravado than he intends (imo). I think he's just pointing out the fact that the latest iteration of his engine IS capable of processing an order of magnitude more triangle data than the current, available version. And the REASON he's being so upfront about this is because the 'old' unreal engine's triangle limitations were far and away the number one complaint people had with the engine. And Tim would never deny that as it was OBVIOUSLY inferior. That one glaring weakness has now been (thankfully) fixed.

[ben6]

It is also an excellent system benchmark, because unlike other benchmarks, the Comanche 4 benchmark is primarily limited by today's CPUs. This is due to the extensive physics, collision detection, AI, and other CPU intensive activities we use in Comanche 4. 3-4GHz CPUs will be required to push the latest generation video cards at lower resolutions on this engine.

from www.nvnews.net

[Humus]

Sound like a way to express a poorly optimized engine as a good thing.

[maguen]

Sound like a way to express a poorly optimized engine as a good thing.

I agree.

That is the problem with closed benchmarks in general. Is it CPU bound because of game representitive A.I., physics, and game logic, or because of poor algorithms and little if any optimization? Is it saturating the AGP bus because of AGP limitations (pushing the envelope) or because of a poorly implemented data architecture?

In my experience, games of similar, if not better, graphic quallity seem to perform much better then Comanche 4 (I own the full version).

To me, a benchmark is only useful if we *really* understand what it is measuring...

Guenther

[pcchen]

I think, it depends on what you are measuring.

IMHO there are basically two types of benchmarks, application benchmarks and component benchmarks. Application benchmarks give the results of running some specific applications. Component benchmarks tries to measure performance of some specific components.

So from this point of view, if one love to play Comanche 4, how it be done does not matter at all. He/she just want to know how it performs on a platform. On the other hand, if one need to know how a video card performs regardless the CPU, He/she may want a synthetic benchmark to exploit its strong side and show its weaknesses.