A new modern engine benchmark

[Oompa Loompa]

It scares me a bit when Tim Sweeney goes on and on about how many polygons Unreal 2 will use. Will it be another game(engine) that will never run particularly well on any hardware in relation to how it looks?

Hmm. Well, Unreal/UT may have never been a good benchmark, but these were good games, and licensees made at least as many good looking and performing UT-engine games as Q3-engine games. So I'm not sure all of your criticism is justified.

But in any case, things seem to be different this time. According to the "Unreal Peformance Test 2002", results seem to scale very much according to GPU power. http://www.anandtech.com/video/showdoc.html?i=1583&p=12

(Hmm. When did Anand start embedding his charts in such a sneaky fashion? As far as I can see the charts don't exist as discrete GIFs or JPEGs, but each bar is a separate object. Makes it hard to steal )

 

[darkblu]

We could probably have a good discussion here about how a large polygon budget should be used though. And what bottlenecks/pitfalls there are. New thread, someone?

that'd be interesting.

 

[Mephisto]

No, the benchmark is clearly CPU limited – mapping out the fillrate graph shows some real oddities with the Radeon 8500 because the drop off at 1600x1200 is far too much. Also the Pixel Shader don’t make much sense as so far all the testing indicates that its actually faster with PS enabled than disabled!

Sure? here with my GF3 on a Thunderbird 1 GHz (<- !) it definitly shows a fillrate limited fall down.

May be you shoud enable the vertex shading on R8500 (off by default ...).

 

[Reverend]

The benchmark itself isn't indicative of gameplay. I have the full retail version and the gameplay really is rather good, being a good mix of straight-out action that many FPS'ers will be familiar with and that of a "sim", a not-so-serious one at that however. That's my opinion of gameplay and as usual gameplay is more subjective than something like IQ.

That said, the IQ is, well, simple. It is nothing to shout about. The only stuff worth mentioning are the rotor water vertex effects (which aren't that realistic anyway) and highish polies. Everything else seem dated to me (look at the explosion effect of the ship when it eventually goes down during the benchmark... it's cartoonish).

The full retail version is like this demo/benchmark when it comes to performance - it is incredibly CPU and bus limited and the explosion effects gobbles up fillrate (which is the primary reason for all the lowest recorded framerates during the benchmark run). I ran a batched benchmark (FYI. I never was able to use the LAUNCH.EXE for batched benchmark so I tried without the LAUNCH.EXE and simply used c4demo.exe for batched benchmarking, which ran fine) :

===========================================

Comanche 4 Benchmark Results for 640x480x32 FSAA=0 VSync=OFF DXTC=ENABLED AUDIO=OFF SHADERS=ENABLED

Frames per second: 40.41 avg
Tris per second: 8,047,552 avg


Comanche 4 Benchmark Results for 1024x768x32 FSAA=0 VSync=OFF DXTC=ENABLED AUDIO=OFF SHADERS=ENABLED

Frames per second: 40.30 avg
Tris per second: 8,025,587 avg


Comanche 4 Benchmark Results for 1600x1200x32 FSAA=0 VSync=OFF DXTC=ENABLED AUDIO=OFF SHADERS=ENABLED

Frames per second: 37.89 avg
Tris per second: 7,546,306 avg


Comanche 4 Benchmark Results for 640x480x32 FSAA=2 VSync=OFF DXTC=ENABLED AUDIO=OFF SHADERS=ENABLED

Frames per second: 40.17 avg
Tris per second: 8,000,690 avg


Comanche 4 Benchmark Results for 1024x768x32 FSAA=2 VSync=OFF DXTC=ENABLED AUDIO=OFF SHADERS=ENABLED

Frames per second: 39.49 avg
Tris per second: 7,863,706 avg


Comanche 4 Benchmark Results for 1600x1200x32 FSAA=2 VSync=OFF DXTC=ENABLED AUDIO=OFF SHADERS=ENABLED

Frames per second: 29.03 avg
Tris per second: 5,782,285 avg


Comanche 4 Benchmark Results for 640x480x32 FSAA=4 VSync=OFF DXTC=ENABLED AUDIO=OFF SHADERS=ENABLED

Frames per second: 39.91 avg
Tris per second: 7,947,273 avg


Comanche 4 Benchmark Results for 1024x768x32 FSAA=4 VSync=OFF DXTC=ENABLED AUDIO=OFF SHADERS=ENABLED

Frames per second: 35.65 avg
Tris per second: 7,098,793 avg


Comanche 4 Benchmark Results for 1600x1200x32 FSAA=4 VSync=OFF DXTC=ENABLED AUDIO=OFF SHADERS=ENABLED

Frames per second: 16.76 avg
Tris per second: 3,338,692 avg


Comanche 4 Benchmark Results for 640x480x32 FSAA=0 VSync=OFF DXTC=ENABLED AUDIO=OFF SHADERS=DISABLED

Frames per second: 38.76 avg
Tris per second: 7,676,165 avg


Comanche 4 Benchmark Results for 1024x768x32 FSAA=0 VSync=OFF DXTC=ENABLED AUDIO=OFF SHADERS=DISABLED

Frames per second: 38.65 avg
Tris per second: 7,655,600 avg


Comanche 4 Benchmark Results for 1600x1200x32 FSAA=0 VSync=OFF DXTC=ENABLED AUDIO=OFF SHADERS=DISABLED

Frames per second: 36.89 avg
Tris per second: 7,307,617 avg


Comanche 4 Benchmark Results for 640x480x32 FSAA=2 VSync=OFF DXTC=ENABLED AUDIO=OFF SHADERS=DISABLED

Frames per second: 38.53 avg
Tris per second: 7,630,637 avg


Comanche 4 Benchmark Results for 1024x768x32 FSAA=2 VSync=OFF DXTC=ENABLED AUDIO=OFF SHADERS=DISABLED

Frames per second: 38.04 avg
Tris per second: 7,534,190 avg


Comanche 4 Benchmark Results for 1600x1200x32 FSAA=2 VSync=OFF DXTC=ENABLED AUDIO=OFF SHADERS=DISABLED

Frames per second: 29.10 avg
Tris per second: 5,764,551 avg


Comanche 4 Benchmark Results for 640x480x32 FSAA=4 VSync=OFF DXTC=ENABLED AUDIO=OFF SHADERS=DISABLED

Frames per second: 38.43 avg
Tris per second: 7,612,581 avg


Comanche 4 Benchmark Results for 1024x768x32 FSAA=4 VSync=OFF DXTC=ENABLED AUDIO=OFF SHADERS=DISABLED

Frames per second: 34.77 avg
Tris per second: 6,885,814 avg


Comanche 4 Benchmark Results for 1600x1200x32 FSAA=4 VSync=OFF DXTC=ENABLED AUDIO=OFF SHADERS=DISABLED

Frames per second: 16.93 avg
Tris per second: 3,353,467 avg

===========================================

That's on a XP2000+, 256MB PC2100, GF4 Ti4600 with official 28.32 drivers, DX8.1, WinXP. Sound was disabled via the game's specified command line for doing so and all AA was implemented via command line option rather than thru NV's display properties. That SHADERS= option is a result of using either "dx7" or "dx8" in the command line. It corresponds to dx7=SHADERS=DISABLED and dx8=SHADERS=ENABLED.

A few notes :
- Note the DX7 vs DX8 scores for all tests, AA or NoAA
- WRT the above, I don't believe the game uses any PS effects, only VS. Accordingly, if I'm correct, DX7 would mean vertex shaders using host CPU and DX8 would mean using vid card supporting VS. I also believe that the VS effects largely has to do with the rotor-on-water effects - and using the command line option to use "DX7" or "DX8" resulted in the same such effect. Note Wavey's comments on this - I think it comes down to which is faster for the VS effects used in this particular game/benchmark - the host CPU or the GPU.
- Note triangle throughput differences at diff rez

This is another game that can be indicative of the way the latest crop of games are being developed - they will become more dependant on the host CPU up until a rez like 1024x768. Stuff like physics, sound and AI will ensure CPU limitation. Look at this particular game, MOHAA, Max Payne...

 

[Humus]

The reality is across DDR and SDR systems, 1gig Athlons to 2 gig pentiums, almost everybody is throwing out similar results.

I suppose that means it's limited by AGP bandwidth. They are probably storing all the geometry in AGP memory, and with that massive polygoncount I suppose it will easily become the bottleneck.

 

[Humus]

I don't think we should complain about Tim Sweeney's work, ...

I disagree.
Though I can certainly forgive the man that he zigged instead of zagged on a particular technical point, I don't much care for his cathegorical statements, judgemental attitude and recently his infantile harping on how Unreal will have 100 000+ polys per frame.
First off, it's no guarantee that it will look particularly good. The Comanche4 Demo makes that embarrassingly obvious. Secondly, it implies that they don't do much (or any) LOD work, making for a gameworld bogged down with unnecessary detail at high distances in order to look reasonably good close up. Your comments about LOD are not without validity of course, but it is not as if it takes a huge amount of work to do, nor does it lhave to look bad, as long as you are not constrained by trying to reduce the scene to an absolute bare minimum number of polygons. Safe, conservative LOD is way better than none at all. Not using it is asking for trouble when you are creating a general engine, or an engine intended for multiplayer use.

Have _anyone_ here seen good framerates in any games when the polycounts climb beyond 6 digits? No, benchmarks such as 3DMarks don't count. Not too many such games around of course, and those that are may not be optimally coded, but.... I think a certain sceptisism is warranted. As long as you can reduce the polycount to improve framerate this won't be a problem, but that reduces all this "over a hundred thousand polygons" to marketing and little else.

We could probably have a good discussion here about how a large polygon budget should be used though. And what bottlenecks/pitfalls there are. New thread, someone?

And I'm sorry, but I can't help but comparing Sweeneys posturing with Carmacks nerdy description of how they make relatively few polys look really good, and how they allocate the available computing budget. it's probably a bit silly of me, but I know which attitude I'm comfortable with.

Entropy

I can agree than Tim's attitude isn't the best, I have a lot more respect for Carmack with regards to that. But he have done some really good work, as have Carmack. I'm very excited about both Unreal2/UT2 and Doom3, but I think Doom3 will look much better.

 

[Randell]

Yes the developers posted that it uses a lot of polygons and is limited by the AGP bus in the thread over at nvNews.

 

[Dave Baumann]

OT Gripe...

Sorry to for the OT gripe in the thread, but....

Could everyone please be a little more circumspect in their Quoting - I know its easy just to hit the quote button and reply, but when you do so please can you trim the quotes to the relavent parts.

You guys are eating database space up at a rapid rate!

 

[Doomtrooper]

Somebody PLEASE leak the UT Technology benchmark, we need a new freakin playtoy and not another review of Quake 3 running at 250 fps..please....

 

[Entropy]

I can agree than Tim's attitude isn't the best, I have a lot more respect for Carmack with regards to that. But he have done some really good work, as have Carmack. I'm very excited about both Unreal2/UT2 and Doom3, but I think Doom3 will look much better.

I apologize for posting while in a grumpy mood, but when doing my gfx-sweep of the web, I first read this interview with Tim Sweeney [http://www.gamespy.com/gdc2002/sweeney/], and then downloaded and ran the Comanche4 demo.

The Comanche4 demo is yet another example of designers using additional graphics performance to do more-of-the-same. We get 100 highly detailed trees in the background rather than 15 cruder ones. Madonion realized that this was a problem with their 3dMark2000, they added their detail in ways that either wasn't percieved at all (i.e. the barrels being smoother) or didn't contribute to scene realism (150 crates and barrels does not make a more realistic dock scene than 20). Their Nature scene in 3DMark2001 on the other hand is an excellent example of gfx power used in an attempt to increase realism in new ways.

Using tons of polygons on static environments solves few actual problems, and bring the potential to create new ones. It is reasonable to assume that at some point in the future, increases in polygon count will level off since it seems wasteful to render multiple polygons per pixel. Our output medium has limited resolution, and textures and bumpmaps can often convey detail information in a way that is convincing. Increases in realism will come from other areas, such as lightning, which has the advantage of being easily expressible in mathematical models even though they may be difficult to implement in an efficient manner on current hardware. More challenging is movement, particularly of characters. Characters and creatures are probably the most difficult, but trees/vehicles/weather/grass/water/cloth et cetera are a bit easier to model, and even crude models are better than nothing. When it comes to characters however I predict that we will soon see 5000-10000 polygon characters that move no more convincingly than 250 polygons of Lara Croft did in the dawn of consumer realtime 3D. And some will probably do worse. This is an example of a problem where we our models are too crude, regardless of how much computing power we throw at the problem. Final Fantasy, the movie, is probably as good as I've seen yet, but their methods may not be easily applicable to real time rendering.

What will definitely not bring us forward one whit is when icons of consumer 3D programming avoid handling terrain LOD and then brag about how their polygon counts have gone up.... going from that interview directly to the Comanche4 demo with its average of 200 000 polygons per frame was a downright depressing example of where such attitudes can take us. Not Very Far At All.

Entropy

 

[Dave Baumann]

OK, I have a responce as to the performance difference between 'DX&' and 'DX8' numbers, I'll post the full responce in the upcoming Radeon 8500 review.

 

[jb]

which will be ready for us to look at when good sir?

 

[Dave Baumann]

With a little luck, later this week/early next week.

 

[Johnny Rotten]

I think you're perhaps being a little harsh towards Mr. Sweeney, or at least interpreting his statements as being more bravado than he intends (imo). I think he's just pointing out the fact that the latest iteration of his engine IS capable of processing an order of magnitude more triangle data than the current, available version. And the REASON he's being so upfront about this is because the 'old' unreal engine's triangle limitations were far and away the number one complaint people had with the engine. And Tim would never deny that as it was OBVIOUSLY inferior. That one glaring weakness has now been (thankfully) fixed.

 

[ben6]

It is also an excellent system benchmark, because unlike other benchmarks, the Comanche 4 benchmark is primarily limited by today's CPUs. This is due to the extensive physics, collision detection, AI, and other CPU intensive activities we use in Comanche 4. 3-4GHz CPUs will be required to push the latest generation video cards at lower resolutions on this engine.

from www.nvnews.net

 

[Humus]

Sound like a way to express a poorly optimized engine as a good thing.

 

[maguen]

Sound like a way to express a poorly optimized engine as a good thing.

I agree.

That is the problem with closed benchmarks in general. Is it CPU bound because of game representitive A.I., physics, and game logic, or because of poor algorithms and little if any optimization? Is it saturating the AGP bus because of AGP limitations (pushing the envelope) or because of a poorly implemented data architecture?

In my experience, games of similar, if not better, graphic quallity seem to perform much better then Comanche 4 (I own the full version).

To me, a benchmark is only useful if we *really* understand what it is measuring...

Guenther

 

[pcchen]

I think, it depends on what you are measuring.

IMHO there are basically two types of benchmarks, application benchmarks and component benchmarks. Application benchmarks give the results of running some specific applications. Component benchmarks tries to measure performance of some specific components.

So from this point of view, if one love to play Comanche 4, how it be done does not matter at all. He/she just want to know how it performs on a platform. On the other hand, if one need to know how a video card performs regardless the CPU, He/she may want a synthetic benchmark to exploit its strong side and show its weaknesses.