QUAKE3 IS A WORTHLESS BENCHMARK. WHY IS IT STILL BEING USED

[rms]

Its now almost 2004, and sites are *still* using this 3-year old game to benchmark.

FACT: Both 3DNOW and SSE are non-functional in Quake3 on AthlonXP cpus (and probably disabled as well on the Athlon64, no data there as yet), but SSE is *enabled* on all Intel cpus. Not a big surprise there; The XP didn't exist when Q3 was programmed. But I *still* see reviewers 'shocked' at how much an P4 outperforms an AthlonXP in Quake3. Helloooo? Time to smell the coffee guys !! What's worse is this same Intel-biased behaviour extends to other Q3-based games.

Talk to this guy for more info, he's quite knowledgable about the history of Quake3 and AMD support. http://speedycpu.dyndns.org/opt/


FACT: Quake3 benchmarks are now hitting the 400-500 fps range. DO THESE KINDS OF SCORES HAVE ANY RELATION TO REALITY. A gamer cannot distinguish the difference between 450 and 500 fps onscreen, and these kinds of scores, not to mention that few people play Q3 anymore, make Q3 irrelevant as a gaming benchmark on modern machines, in particular when comparing modern cpus.


THE POINT IS: Quake3 is a old, biased and irrelevant benchmark. Its use should be DISCONTINUED in reviews, and in particular Intel vs AMD cpu reviews.

rms

 

[Tagrineth]

Re: QUAKE3 IS A WORTHLESS BENCHMARK. WHY IS IT STILL BEING

FACT: Both 3DNOW and SSE are non-functional in Quake3 on AthlonXP cpus (and probably disabled as well on the Athlon64, no data there as yet), but SSE is *enabled* on all Intel cpus. Not a big surprise there; The XP didn't exist when Q3 was programmed. But I *still* see reviewers 'shocked' at how much an P4 outperforms an AthlonXP in Quake3. Helloooo? Time to smell the coffee guys !! What's worse is this same Intel-biased behaviour extends to other Q3-based games.

Talk to this guy for more info, he's quite knowledgable about the history of Quake3 and AMD support. http://speedycpu.dyndns.org/opt/

Actually, Carmack's response when asked about P4's extreme performance in Q3 was something more along the lines of 'Hmm, not sure... might be because the engine is modular and fits in P4's cache structure well'.

FACT: Quake3 benchmarks are now hitting the 400-500 fps range. DO THESE KINDS OF SCORES HAVE ANY RELATION TO REALITY. A gamer cannot distinguish the difference between 450 and 500 fps onscreen, and these kinds of scores, not to mention that few people play Q3 anymore, make Q3 irrelevant as a gaming benchmark on modern machines, in particular when comparing modern cpus.

That argument has been around since Q3 reached ~150fps. No big news.

The thing is, Q3 is still highly responsive to changes in hardware configuration. Any changes in your system WILL be reflected in Q3's frame rate. That's why Q3 is still used - it isn't useful for absolute performance, it's useful for relative DX7 performance.

 

[GraphixViolence]

Re: QUAKE3 IS A WORTHLESS BENCHMARK. WHY IS IT STILL BEING

Actually, I believe Q3 Arena was released in late 1999, so that would make it about 4 years old. The release lined up nicely with the GeForce 256, the Rage Fury MAXX, and the Savage 2000, all of which could run it at 30-50 fps at 1024x768 with no AA or AF. Good to know the benchmarks are keeping up with the hardware

 

[rms]

Re: QUAKE3 IS A WORTHLESS BENCHMARK. WHY IS IT STILL BEING

Actually, Carmack's response when asked about P4's extreme performance in Q3 was something more along the lines of 'Hmm, not sure... might be because the engine is modular and fits in P4's cache structure well'.

I was told (I have no way to verify this) that AMD approached Carmack with a request to add AthlonXP support to Quake3, and that he refused for whatever reason. All that would be needed to even the playing field is to recognize the AthlonXP and beyond cpus as SSE-enabled.

That's why Q3 is still used - it isn't useful for absolute performance, it's useful for relative DX7 performance.

When a benchmark does not support a major feature of the cpu it is being used to benchmark while enabling that IDENTICAL feature on a competing cpu, it is UNFAIR and should not be used to compare the cpus. What is your argument against this statement?

rms

 

[jpaana]

Re: QUAKE3 IS A WORTHLESS BENCHMARK. WHY IS IT STILL BEING

FACT: Both 3DNOW and SSE are non-functional in Quake3 on AthlonXP cpus (and probably disabled as well on the Athlon64, no data there as yet), but SSE is *enabled* on all Intel cpus. Not a big surprise there; The XP didn't exist when Q3 was programmed. But I *still* see reviewers 'shocked' at how much an P4 outperforms an AthlonXP in Quake3. Helloooo? Time to smell the coffee guys !! What's worse is this same Intel-biased behaviour extends to other Q3-based games.

Talk to this guy for more info, he's quite knowledgable about the history of Quake3 and AMD support. http://speedycpu.dyndns.org/opt/

If you examine a stock Quake 3 installation, you'll see that it doesn't contain any dlls, only .qvm files (inside the .pk3s), which are the same game code this guy "optimized" but for the virtual machine which is used to run the code. This doesn't include any of the rendering code for example, only the public game logic and menu code, but parts of it are run during timedemos as demos are basically just dump of the network data (run in the same machine or elsewhere). Running the virtual machine code is much more memory intensive than running the native code as it doesn't contain any JIT compiler and thus pushes lots of data to memory and back, thus P4 having more FSB and memory bandwidth usually wins. Running native code can use registers and balances the score considerably, plus gives a nice speed boost on both processors.

The reason for the virtual machine being used is that the same code can be run regardless of the user running Windows, Linux, Mac or what ever and thus mods for example don't have to release different versions for each.

 

[rms]

Re: QUAKE3 IS A WORTHLESS BENCHMARK. WHY IS IT STILL BEING

[quote="jpaanaIf you examine a stock Quake 3 installation, you'll see that it doesn't contain any dlls,....[/quote]

Quite right. I guess I did not make the point clear.

I am not advocating the use of these dlls (they don't fix the main problem of AthlonXP SSE non-detection anyway).

Nor am I asking for Carmack to make a patch for Quake3. He won't.

I am saying Stop Using Quake3 As A Benchmark To Compare AMD vs Intel Cpus In Reviews. It is unfair, invalid, and irrelevant.

rms