Q3 as benchmark

[pascal]

I am starting this topic because I dont want to continue what can potentially be a long OT in the other thread (http://216.12.218.25/domain/www.beyond3d.com/forum/viewtopic.php?t=623)

Democoder wrote:
People won't stop using Q3 as a benchmark until someone else ships a game that scales with CPU, bus speed, and video card speed. Q3 seems to be engine that shows significant improvement on much faster CPUs, or DDR/RDRAM systems, or on fast cards.

If you run a game that is locked at 40fps because of some bad coding or bottleneck, and runs the same on a 500 Mhz Celery + TNT2 as it does on a P4+GF3, it's obviously not a good benchmark.

Some of the other engines are getting better, but I still haven't seen any engine that scales so beautifully with cpu, bus, and vid card.

Mark wrote:
But doesn't this in itself make its use a poor way of evaluating real-world gaming performance? I mean, if you took the top twenty selling 3D games and ran them on a GF2 and on a GF4 and the framerate is about the same, doesn't that say the speed of the videocard at that level is irrelevant? Didn't UT tell us that it didn't matter if you were running a Voodoo3 a GF2 Ultra because the framerate was about the same? So image quality and featureset mattered more than raw speed? Or that CPU speed mattered more than videocard speed?

I think that might be part of the problem with such broad use of Q3 benchmarking - it doesn't really relate to real-world videocard use, doesn't address PC hardware balance issues very well since CPU scaling went by the wayside, and drives the manufacturers to emphesize speed and framerate (particularly in this game - look at the initial 8500 thing) over quality and driver and game compatability and development. It's not even a real-world test anymore, rather is a theoretical test - who can really tell the visual difference between the 220fps a Ti 4600 can make vs. the 150 or so an MX460 can do?
_________________
Choosing a video card is like choosing a hooker - go for the most bang for the buck...

When I said that "People still use Q3 for benchmark" I was trying to say that not much changed in the gaming world. We are still waiting the next great gaming engine.

Q3 is a good benchmark and I think people can and should still use it, but not only it.

It is good to see the relative performance between similar components:
-See the difference between the different P4s
-See the difference between P3 and Athlons.
-See the fillrate capability of 3D cards at high resolution.
-See the difference between memory or mobos.

But it is hard to predict the performace of other games using only it.
Also it doesnt test the situations below:
a) Extreme high polygons situations with hw T&L
b) Advanced multitexturing with per pixel lighting

For example, I like very much the 3Dmark2001 car chase high detail for CPUs/mobo/memory performance prediction (any CPU) with the situation a) . It is an extreme benchmark with linear increase for +1GHz CPUs and my guess anyone with a 30fps or better will have a good gameplay with some future hw T&L games.

Then I am not saying "dont use Q3", I am trying to say dont use only Q3.

 

[Entropy]

Apart from some severely warped individuals, (some of whom reside on this forum , people benchmark for the purpose of gaining information, not entertainment. Just as Democoder stated, Q3 can provide pertinent data on the performance of all parts of the CPU/memory system/gfx-card triangle. Apart from being a game that is played by a large number of people of whom many actually care about the performance of their game, the engine has been used for a number of hit games, and continues to be so used since it can provide good performance on a wide range of hardware in addition to great visuals. Plus of course, quite a bit of experience in its use has accumulated.

By now there are gfx-features that cannot be tested by Q3, mainly vertex and pixel shaders. Personally, since I haven't seen any remotely interesting game that depends on either, I'm happy using a synthetic benchmark like 3DMark2001SE for some rough idea on the progress being made in these areas.

For a game benchmark to have any particular valueto a review, it must either feature a game(engine) that is very widely used (since then the information presented is directly applicable for prediction purposes for a reasonable portion of readers), or it must be typical enough that the results have predictive value for games based on other engines. Unless either of these conditions are fulfilled, there is little reason to bother with games benchmarks at all. Stick with synthetic tests, since informed readers can dig into that data for predictive purposes, and less informed readers will enjoy that glittering single figure of merit that these benchmarks typically produce at the end of their run.

Us graphics weenies should take a step back and take note that the overwhelmingly most popular online FPS still uses the Quake2 engine! And looking at the list over the hottest selling games hardly supports the opinion that we desperately need good games for testing shaders since you could argue that most of the bestsellers don't even use 3D graphics. Please note the complete lack of Aquanox on any such list. Until DOOM3 or possibly Unreal2 ships, I can't see any new game that is of general interest to test. Perhaps not even then, since either of these may display anomalous behaviour making them unsuitable for general purpose benchmarking, such as UnrealTournament always has been.

Doing "spot-tests" with different games is of course valid, and can serve to remind people that different games have different profiles. But as far as general validity/interest is concerned, the outlook is grim.

Entropy

 

[Doomtrooper]

The problem is no one plays Quake 3 online and in fact never was a big online game. The reason why was simple, it sucked. Pretty engine and terrible game play. According to Ngstats there were as much as three times as many Unreal Tournament players online vs Quake 3.
Another example is counterstrike, very old engine but good gameplay.
I have a big issue with Quake 3 as a benchmark, one its very INTEL optimised with SSE instructions out the Ying Yang. The other major factor is its OLD, three years old now
Newer games based off a tweaked Quake 3 which contain more modern features like lighting improvements etc. would be fine and I see Return to Castle Wolfenstein a good example.
Serious Sam 2 is another good example of a modern engine with LOTS of outdoor scenes.

Now my main concern about benchmarking games like Quake 3, instead of running some time demo that runs you through a hallway and you fight a few bots I'D MUCH rather see the reviewer jump ONLINE to a actual server and play a few matches with 20 + other players running around. This is the real deal test and much more realistic, because in my experience in online games (and I play alot of them) the time demo is not accurate way of benchmarking with high quality settings in realistic way. There is no way any card could run 1024 x 768 @ 32 bit 4 x FSAA with Max Anistropic on MOHAA online and get playable frame rates.
Anyone that has played online knows anything below 20 fps makes for a very poor gaming experience, very hard to hit your opponent.
If you ran a time demo with these settings though, you'd probably be ok yet this isn't truly a REAL world test for a online game.

 

[PVR_Extremist]

Good points.

To be honest, I place more value on the reviewers overall experience with a card than plain numbers. Virtually every card you can buy these days handles Q3 very well.

Compatibility issues, dodgy drivers, visual anomolies, overall image quality etc these are just as important as speed.

I then weigh those criteria up against value for money as this to me is of utmost importance. Sure, everyone wants to have the best card in their machine but it's just not a reality for most of us.

Most of us have to settle for the best we can afford given our requirements.

 

[Nil Einne]

Online test?

I personally don't think an online test is that great an idea. Firstly, offline is still a lot more popular then online. But maybe much more importantly, the kind of test you mentioned IMHO has too many uncontroable variables to be of any issue. The fact is performance can vary greatly day by day. E.g., today is a quite day and so everything is very fast but then tomorrow Doom 3 demo comes out and your ISP is overwhelmed. Or maybe they have a bad ISP or are still using dialup or whatever and so performance varies significantly. Even worse, a person's enjoyment of the game depends a lot on how things go. So for example, maybe one day you meet a bunch of losers but tomorrow you meet the friends who've been playing the games 18/7 for the past 3 years and so get massacred. The reviewer may end up blaming the card (or at least getting a lower opinion of) the card knowingly or unknowingly. I'm not saying it's impossible, it's possible but the time required is too much IMHO for the little info gained. Note it's very subjective too.........

 

[Nil Einne]

Not best card that fits budget IMHO

I'd think it's more price/performance (performance not in absolute terms but more of performance in terms of likely personal experience with the card, how long it will last before an upgrade is required, whether we actually need the extra experience or perfomance etc) but taking into account budget issues.

That's why people, e.g. me can consider both the MX440 and the Ti200 and the Radeon8500LE. Is the Ti200 worth the extra price? What about the 8500LE?

 

[Saem]

The problem is no one plays Quake 3 online and in fact never was a big online game. The reason why was simple, it sucked. Pretty engine and terrible game play. According to Ngstats there were as much as three times as many Unreal Tournament players online vs Quake 3.

well according to gamespy that's not the case, there are usually more people playing Q3 than UT, of course that includes mods, but still, the q3 engine is being put to use.

Another example is counterstrike, very old engine but good gameplay.

Get a clue, CS is a quake engine game, it has very simillar performance characteristics to other quake engine games, Q3 benches are telling of the performance you'll see in CS, but performance really isn't an issue in the game.

I have a big issue with Quake 3 as a benchmark, one its very INTEL optimised with SSE instructions out the Ying Yang.

So what? Do you realize that Q3 is first a game and then a benchmark, just because it has SSE optimizations is perhaps more telling of the fact that Carmack felt they were needed and were worth the effort. BTW, Intel spends a lot of money helping developers, this is merely return on investment, as for AMD, if they had much developer support to speak of in those days perhaps we'd see AMD centric optimizations, not that 3DNow! would yeild much improvement for the K7 and the K6's FP capabilities might just have been too weak to help.

Serious Sam 2 is another good example of a modern engine with LOTS of outdoor scenes.

Yes and guess what, just like Q3 this game spends >50% of it's time in the OGL driver, according to Carmack Q3 spends 70% of it's time there, it's a big test for OGL drivers. The problem is SS, unlike Q3 is a very different beast. Q3 isn't dynamic, as in it doesn't turns on and off features for different cards unlike Serious Sam which does. If you use one of those scripts which locks the settings you're not really revealling what it runs like on your machine, unless you use the scripts. Basically, you need an engine which uses graphics cards features and doesn't have any LOD adjusting so every card renders the EXACT same thing. But then it wouldn't be a real world benchmark.

 

[jb]

Get a clue, CS is a quake engine game, it has very simillar performance characteristics to other quake engine games, Q3 benches are telling of the performance you'll see in CS, but performance really isn't an issue in the game.

Not really true. CS seems to be more of a CPU limited game. We have seen nVidia at the top of the Q3 crown and yet in RtCW at some settings the 8500 is faster (slightly).

 

[Livecoma]

I don't believe any other benchmark produces results as consistant as Quake3 has. The way it stresses the entire system at a given resolution is just good for benchmarking, IMO.

 

[Doomtrooper]

The problem is no one plays Quake 3 online and in fact never was a big online game. The reason why was simple, it sucked. Pretty engine and terrible game play. According to Ngstats there were as much as three times as many Unreal Tournament players online vs Quake 3.

well according to gamespy that's not the case, there are usually more people playing Q3 than UT, of course that includes mods, but still, the q3 engine is being put to use.

Another example is counterstrike, very old engine but good gameplay.

Get a clue, CS is a quake engine game, it has very simillar performance characteristics to other quake engine games, Q3 benches are telling of the performance you'll see in CS, but performance really isn't an issue in the game.

I have a big issue with Quake 3 as a benchmark, one its very INTEL optimised with SSE instructions out the Ying Yang.

So what? Do you realize that Q3 is first a game and then a benchmark, just because it has SSE optimizations is perhaps more telling of the fact that Carmack felt they were needed and were worth the effort. BTW, Intel spends a lot of money helping developers, this is merely return on investment, as for AMD, if they had much developer support to speak of in those days perhaps we'd see AMD centric optimizations, not that 3DNow! would yeild much improvement for the K7 and the K6's FP capabilities might just have been too weak to help.

Serious Sam 2 is another good example of a modern engine with LOTS of outdoor scenes.

Yes and guess what, just like Q3 this game spends >50% of it's time in the OGL driver, according to Carmack Q3 spends 70% of it's time there, it's a big test for OGL drivers. The problem is SS, unlike Q3 is a very different beast. Q3 isn't dynamic, as in it doesn't turns on and off features for different cards unlike Serious Sam which does. If you use one of those scripts which locks the settings you're not really revealling what it runs like on your machine, unless you use the scripts. Basically, you need an engine which uses graphics cards features and doesn't have any LOD adjusting so every card renders the EXACT same thing. But then it wouldn't be a real world benchmark.

If IT'S a online GAME which QUAKE 3 is it should be tested as such, I don't KNOW anyone that played the single mission Quake 3 because there isn't any.
I don't see how benchmarking in a real world test has anything to do with the ISP or maybe you had better get a new one.
I'm on DSL and I can pull up Unreal Tournament or MOHAA and jump on my favorite servers and get the same ping and frames per second consistently. I seriously don't think its very difficult and will result in a real world test.

 

[Doomtrooper]

The problem is no one plays Quake 3 online and
So what? Do you realize that Q3 is first a game and then a benchmark, just because it has SSE optimizations is perhaps more telling of the fact that Carmack felt they were needed and were worth the effort. BTW, Intel spends a lot of money helping developers, this is merely return on investment, as for AMD, if they had much developer support to speak of in those days perhaps we'd see AMD centric optimizations, not that 3DNow! would yeild much improvement for the K7 and the K6's FP capabilities might just have been too weak to help.

I have a REAL problem with optimized benchmarks for particular hardware. If a CPU clocked the same can give a speed advantage over the other on a video card benchmark test due to CPU OPTIMIZATIONS then that kinda throws the Video card speed numbers down the toilet doesn't it. You're not really testing video card speed, but how fast a Pentium 3 and 4 will go with a certain video card

Serious Sam is heavily AMD optimized so I guess the same could be said for that benchmark.

http://www.anandtech.com/showdoc.html?i=1574&p=12


You see what 3Dnow 2 Optimizations can also do, but now we have a CPU benchmark vs a video benchmark

 

[OpenGL guy]

CS != RtCW

Get a clue, CS is a quake engine game, it has very simillar performance characteristics to other quake engine games, Q3 benches are telling of the performance you'll see in CS, but performance really isn't an issue in the game.

Not really true. CS seems to be more of a CPU limited game. We have seen nVidia at the top of the Q3 crown and yet in RtCW at some settings the 8500 is faster (slightly).

Half-life (the game upon which Counter-Strike runs) is using the Quake 2 engine, with some enhancements. It was originally based on the Quake 1 engine, but it was upgraded to Quake 2 before release.

RtCW is using the Quake 3 engine.

You cannot compare CS performance to RtCW performance since the games and their engines are completely different. Secondly, I wouldn't say that CS is even remotely CPU limited since the geometry and other aspects of the game are very simplistic. In fact, I wouldn't even recommend using CS as a benchmark at all, even though I enjoy the game thoroughly. My main reason for this belief is that the game is using such an old engine that doesn't use any modern features... and uses some that aren't very popular anymore (i.e. paletted textures).

Quake 3 isn't the most video intensive benchmark around (which is why you have to crank up the resolution or add AA to really differentiate between cards), but saying that you can't evaluate it without playing online is crap. The point is to test video card/CPU/memory performance. If you want to add in more variables (network latency, packet loss, server load) then you are really going to confuse the issue.

 

[Doomtrooper]

Re: CS != RtCW

Get a clue, CS is a quake engine game, it has very simillar performance characteristics to other quake engine games, Q3 benches are telling of the performance you'll see in CS, but performance really isn't an issue in the game.

Not really true. CS seems to be more of a CPU limited game. We have seen nVidia at the top of the Q3 crown and yet in RtCW at some settings the 8500 is faster (slightly).

Half-life (the game upon which Counter-Strike runs) is using the Quake 2 engine, with some enhancements. It was originally based on the Quake 1 engine, but it was upgraded to Quake 2 before release.

RtCW is using the Quake 3 engine.

You cannot compare CS performance to RtCW performance since the games and their engines are completely different. Secondly, I wouldn't say that CS is even remotely CPU limited since the geometry and other aspects of the game are very simplistic. In fact, I wouldn't even recommend using CS as a benchmark at all, even though I enjoy the game thoroughly. My main reason for this belief is that the game is using such an old engine that doesn't use any modern features... and uses some that aren't very popular anymore (i.e. paletted textures).

Quake 3 isn't the most video intensive benchmark around (which is why you have to crank up the resolution or add AA to really differentiate between cards), but saying that you can't evaluate it without playing online is crap. The point is to test video card/CPU/memory performance. If you want to add in more variables (network latency, packet loss, server load) then you are really going to confuse the issue.

Hardly crap...

I've played online games for years, professional level team matches and there is no replacement for HUMAN AI.
Sure some time demo runs you through a set path with say 8 bots ( I think that is the most Quake 3 will allow) that was pre-recorded probably from some game coder. The bots do not react like humans and some servers have as MANY as 30 players.
Now your single player settings are quite different, as you move through the levels you face 5-6 BOTS not 29.
So whatever settings you were using in the single player will not be playable online.
I play Unreal Tournament professionally, I have for two years..I am ranked # 34 in the world and I was in the top 10 but don't play near as often. My settings that I play with in single player 1600 x 1200 Max Anistropic on my 8500 are not playable online.
The server I play on has a 20 player cap, you get 19 players chasing/protecting you in Capture the Flag there is no way a time demo can simulate this..and this IS REALALITY

 

[Fred]

Doom, we're into benchmarking video cards. Not the entire system.

Sure its impossible to gauge what realworld performance is, using just a benchmark that stresses a video card (since its only a part of a whole), but the idea is only relative.

Ie one card is better than another in such and such a situation. Presumably, we hope, it stays that way for *similar* but not identical situations.

As far as Q3 is concerned. It should still be used (especially at high res, or with FSAA cranked), (and as a test of ogl drivers at low res), but yea its time as the best gauge for fillrate has passed.

A new bench (synthetic or other) with more texture passes, higher geometry loads, Overdraw, etc is necessary. A few decent examples are already out there now.

 

[OpenGL guy]

Re: CS != RtCW

[quote="Doomtrooper]
Hardly crap...

I've played online games for years, professional level team matches and there is no replacement for HUMAN AI.
Sure some time demo runs you through a set path with say 8 bots ( I think that is the most Quake 3 will allow) that was pre-recorded probably from some game coder. The bots do not react like humans and some servers have as MANY as 30 players.
Now your single player settings are quite different, as you move through the levels you face 5-6 BOTS not 29.
So whatever settings you were using in the single player will not be playable online.
I play Unreal Tournament professionally, I have for two years..I am ranked # 34 in the world and I was in the top 10 but don't play near as often. My settings that I play with in single player 1600 x 1200 Max Anistropic on my 8500 are not playable online.
The server I play on has a 20 player cap, you get 19 players chasing/protecting you in Capture the Flag there is no way a time demo can simulate this..and this IS REALALITY[/quote]

Ok, let me try again.

A benchmark is designed to give a common test environment to compare relative performance of different subsystems. This test environment may or may not match your environment. Now, since you play UT professionally, then you should know that the game is heavily CPU dependent, for example.

How does bot reaction affect a benchmark? It shouldn't. As long as you are drawing a player model, it has no effect whatsoever whether it's a bot or a human controlling the model.

So, here's what you do: Look at a benchmark to see the results in a fixed environment then take into account your own environment. In your case, of course having 30 players around is going to affect your performance. Does that make testing an 8 (or whatever) player game any less valid? No. It does mean that you will need to adjust your settings in order to get good results. Hey, as you said, this is REALALITY(sic).

P.S. What does your ranking in UT have to do with video card performance? None that I can see.

 

[Doomtrooper]

My ranking shows I'm not talking out my ASS and know what kind of frames are required for professional play

Games that are mainly played online like Unreal Tournament and Quake 3 are benched with 8 bots. These bots don't load up 5 rockets in the rocket launcher or try to do double's with the shock rifle !! The difference between BOTS and HUMANS is human players do things no programmer could ever code in, be it fill the room full of FLAK from the flak gun while the other 4 players trying to put 20 rockets through you.
To add to this you have 19 player models to draw, some protecting you with their weapons. So the video card may have to draw as many as 12 or 15 models all firing weapons in the same room.
My main issue I see with games like Quake 3 that a reviewer will state, well I ran 1600 x 1200 32 bit 4x FSAA and Anistropic and XX frames per second. These settings can't be used online and we need some way of testing games that have massive muliplayer capability with as many as 20 plus models on the screen firing, flying etc..

I guess the real issue for me is get a real world result.

A benchmark is designed to give a common test environment to compare relative performance of different subsystems. This test environment may or may not match your environment

The flaw in that statement is a benchmark is designed to give a common test enviroment and QUAKE 3 being a online ONLY game my enivroment example is the real one. The common test enviroment being used is simply not correct.

 

[Saem]

I have a REAL problem with optimized benchmarks for particular hardware. If a CPU clocked the same can give a speed advantage over the other on a video card benchmark test due to CPU OPTIMIZATIONS then that kinda throws the Video card speed numbers down the toilet doesn't it.

First of all, clock speed isn't a means to say two CPUs is equal -god how many times does this have to be said. As for the optimizations, they're mainly in the sound engine. Regardless, your point isn't valid, one could bring up points that Carmack may have heavily optimized for certian video cards leaving others out. The fact of the matter is that it's a game first benchmark second. But it's a really good benchmark because it's accurate and it can be made to heavily test the OGL ICD of a video card.

Serious Sam is heavily AMD optimized so I guess the same could be said for that benchmark.

No, it's not. Bother doing some research rather than talking out of your ass. On Croteam's website, they said that they didn't do any SSE or 3DNow! optimizations. They heavily use MMX for sound mixing in SS. The reason the Athlon does well is because as they (Croteam) said, SS uses the FPU heavily, the Athlons have a very nice FPU, there are no optimizations, however.

 

[LeStoffer]

I don't believe any other benchmark produces results as consistant as Quake3 has. The way it stresses the entire system at a given resolution is just good for benchmarking, IMO.

I concur. Quake 3 is still useful to gauge the overall graphics system (including CPU and main Ram bandwidth). You just need to add other benchmarks to pinpoint the strong and weak points of a particularly card (e.g. fillrate, bandwidth problems, pixel shader performance, vertex shader performance, FSAA performance etc.)

Quake3 is still a good basic benchmark because a lot of games are based on that engine. You (the reviewers that is) just need to add more flavours.

 

[Doomtrooper]

I have a REAL problem with optimized benchmarks for particular hardware. If a CPU clocked the same can give a speed advantage over the other on a video card benchmark test due to CPU OPTIMIZATIONS then that kinda throws the Video card speed numbers down the toilet doesn't it.

First of all, clock speed isn't a means to say two CPUs is equal -god how many times does this have to be said. As for the optimizations, they're mainly in the sound engine. Regardless, your point isn't valid, one could bring up points that Carmack may have heavily optimized for certian video cards leaving others out. The fact of the matter is that it's a game first benchmark second. But it's a really good benchmark because it's accurate and it can be made to heavily test the OGL ICD of a video card.

Serious Sam is heavily AMD optimized so I guess the same could be said for that benchmark.

No, it's not. Bother doing some research rather than talking out of your ass. On Croteam's website, they said that they didn't do any SSE or 3DNow! optimizations. They heavily use MMX for sound mixing in SS. The reason the Athlon does well is because as they (Croteam) said, SS uses the FPU heavily, the Athlons have a very nice FPU, there are no optimizations, however.

Hmmm I think I read it on Anandtech years ago that 3Dnow optimization was the reason for the speed increase. Oh well I did some research and your correct, there appears to be no 3Dnow optimizations in the engine but at the video card driver level.
My point was the Quake 3 engine will give different results with different processors which is not a ideal for benchmarking.
As for the MHZ rating, that is completely useless and undervalued rating system anymore. If the MHZ rating was a acurrate way of measuring speed then there should be no way a 1.5 GHZ CPU should beat a 1.8 GHZ CPU

 

[Humus]

Re: CS != RtCW

I've played online games for years, professional level team matches and there is no replacement for HUMAN AI.

Human AI?
Human artificial intelligense, what would that be, a stupid guy pretending he's smart?

Anyway, Doom, the reason we need to have a fixed environment for benchmarking is that we don't want randomness in the equation. If we benchmark our games in online gameplay, then we get loads of randomness. If we have randomness in the results we will need some form of statistical analysis to sort it out, which means we will need lots of samples, that is, we need to rerun the benchmark tens of times to get meaningful results.
Playing at say CTF-Face it's possible to end up with a average of > 150fps one match, and < 50 fps one the next, depending on how you play. Camping in the base you'd easily get 150fps average, but a shockwhore in the middlefield can easily get below 50fps on average with the same settings.