Use of Custom Demo's In Reviews

Dave Baumann

Gamerscore Wh...
Moderator
Legend
We've talked about using custom demos in reviews since this 3DMark stuff blew up. Brandon Bell over at FS has done just that with both Quake 3 and SS:SE - normally these reside in the playground of NVIDIA, but look at the order of the scores when a non-standard benchmark is used:

http://firingsquad.gamers.com/hardware/msi_geforce_fx5900-td128_review/page8.asp

http://firingsquad.gamers.com/hardware/msi_geforce_fx5900-td128_review/page9.asp

Not exactly the orders we'd normally expect to see for these two benchmarks.
 
It's funny. I remember bringing up the possibility of Nvidia doing the same kind of driver "optimizations" for games that have timedemo systems, and being laughed at by some people for such a preposterous notion.

Well, my purchase decision has been made. Definitely voting with my wallet on this one.
 
Dave,

besides the custom benchmarks, the following differences there are between this review and the last one they did:

- Serious Sam SE is run in quality mode instead of normal mode.
- Quake III is run with the latest point release instead of the version they normally use for testing.
- Quake III is run at 4x FSAA and 16x anisotropic filtering while their previous review didn't use those settings at all (as far as I can see).

So, the swapped places may actually be a result of the different settings than the custom demo. The Radeon 9700 against the Geforce FX always way much stronger in high quality settings in games.

Besides the differences there can be many differences between the standard demos and the ones they recorded, so different areas of the GPU might be differently stressed.
 
Natoma said:
It's funny. I remember bringing up the possibility of Nvidia doing the same kind of driver "optimizations" for games that have timedemo systems, and being laughed at by some people for such a preposterous notion.

Well, my purchase decision has been made. Definitely voting with my wallet on this one.

Your idea wasn't shot down by many as I have also stated that nVIDIA WAS optimising for timedemos. I also had thought it was pretty obvious and that everyone had already known. Apparently not.
 
I think this became obvious atleast to me when I saw the review over at Nvnews. Mike C. used Fraps and non-standard games and the 5900U scores weren't very impressive. Then if you look at other non standard games from other reviews like the NOLF2 getting beat by a 9700 pro it becomes quite obvious.
 
- Quake III is run at 4x FSAA and 16x anisotropic filtering while their previous review didn't use those settings at all (as far as I can see).
Didn't know that Nvidia supported 16x AF... :oops: ;)

Anyway, I think you have a point... We need more tests to be sure that the scores are due to Nvidia "optimizing"...
 
Tokelil said:
- Quake III is run at 4x FSAA and 16x anisotropic filtering while their previous review didn't use those settings at all (as far as I can see).
Didn't know that Nvidia supported 16x AF... :oops: ;)

Anyway, I think you have a point... We need more tests to be sure that the scores are due to Nvidia "optimizing"...

Do we have reason to believe they are not optimising?
With the recent Futuremark events I am sure many people distrust nVIDIA to the point where any high scores with their hardware automatically results in them "optimising" and I for one am not surprised if that is the case 100% of the time.
 
I just put a link to this thread in nvnews and automatically get erase.
Why? Have just been moved to K.I.L.E.R thread. (http://www.nvnews.net/vbulletin/showthread.php?s=&threadid=13456)

This is what I wrote:

Interesting numbers for quake3 and SS:SE have been shown
by Brandon Bell at FS.



Writen by DaveBaumann in beyond3d
http://www.beyond3d.com/forum/viewtopic.php?t=6459

quote:
We've talked about using custom demos in reviews since this 3DMark stuff blew up. Brandon Bell over at FS has done just that with both Quake 3 and SS:SE - normally these reside in the playground of NVIDIA, but look at the order of the scores when a non-standard benchmark is used: http://firingsquad.gamers.com/hardw...eview/page8.asp http://firingsquad.gamers.com/hardw...eview/page9.asp Not exactly the orders we'd normally expect to see for these two benchmarks.
 
Ouchers for nVidia.

I'm not really familar with firingsquad or the gentleman who did the review, but the numbers look pretty consistant to me and like whoever did it knew what they were doing....could someone vouch for the place?
 
Read my post that I have made. :)
It was near yours.


jpeter said:
I just put a link to this thread in nvnews and automatically get erase.
Why?

This is what I wrote:

Interesting numbers for quake3 and SS:SE have been shown
by Brandon Bell at FS.



Writen by DaveBaumann in beyond3d
http://www.beyond3d.com/forum/viewtopic.php?t=6459

quote:
We've talked about using custom demos in reviews since this 3DMark stuff blew up. Brandon Bell over at FS has done just that with both Quake 3 and SS:SE - normally these reside in the playground of NVIDIA, but look at the order of the scores when a non-standard benchmark is used: http://firingsquad.gamers.com/hardw...eview/page8.asp http://firingsquad.gamers.com/hardw...eview/page9.asp Not exactly the orders we'd normally expect to see for these two benchmarks.
 
jpeter: The reason is that nvidia really took some lumps with their cheating on FutureMark and now it is being revealed that they are doing much the same "optimizations" in game benchmarks. The deletion of that post on the nvidia fan site "nvnews" sounds like damage control. If word spreads that nvidia is doing benchmark "optimizations" for games as well then it may result in the same sorts of accusations that FutureMark implied with their test, and that would be a very bad outcome for nvidia. Anyhow, not that you didn't realize why your thread was deleted so quickly, that is the reason, IMO of course.
 
Ok I am having some problems with some of the other numbers in this review like these

http://firingsquad.gamers.com/hardware/msi_geforce_fx5900-td128_review/page12.asp

I simply do not believe that the Nvidia card is really doing AA+AF on this game. There is just no freking way that the Fx is going to be 39 FPS faster than the 9800pro with AA+AF applied.

I repeat.. NO WAY.

I find a couple other oddities in the numbers as well. Some of it can be accounted for becuase of the Nvidia cards Core Speed advantage. But..
 
Sabastian said:
jpeter: The reason is that nvidia really took some lumps with their cheating on FutureMark and now it is being revealed that they are doing much the same "optimizations" in game benchmarks. The deletion of that post on the nvidia fan site "nvnews" sounds like damage control. If word spreads that nvidia is doing benchmark "optimizations" for games as well then it may result in the same sorts of accusations that FutureMark implied with their test, and that would be a very bad outcome for nvidia. Anyhow, not that you didn't realize why your thread was deleted so quickly, that is the reason, IMO of course.

His post was merged with mine as it was about the exact same thing. :)
 
FYI : Scott Sellers, when 3dfx was still around, wrote Kristof, Dave Barron and me the following, just after the release of that infamous 6.16 drivers back in Aug'2000 (GF2 GTS days) :

...so we got our hands on them and ran 'em. Here are the results we found,
along with the results from Radeon (from sharkyExtreme):

<<Nvidia 6.16 vs ATI Quake3.xls>> ...is it me or is it just coincidence that the new 6.16 drivers give the GTS
just enough performance to put it over the top in the 32bpp numbers where
ATI used to have an advantage? Maybe I believe in conspiracies, but
something smells bad about this whole thing. Of course I'm biased, but I
just find this whole thing very suspicious. Plus I just don't trust nvidia.
Has anyone:
- Run the new 6.16 drivers using a different demo set than the
"usual" ones (demo001, demo002, Quaver, etc.)? Do the 6.16 drivers see
benefit across the board on all demo sets, or are they just optimized for
the ones reviewers typically use?
- Has anyone actually measured real-world performance benefit? The
ideal thing to do here would be to put 2 identical GTS systems side by side,
one with 6.16 drivers and one with the older (5.22) drivers and then put one
of them in "follow" mode on the network...turn on the framerate counter in
both and run around a real-world gameworld...is there real-world perf.
benefit using the new drivers (at high rez, 32bpp cases which is where they
seem to have made the most perf. increase)? If not, then they have just
optimized the benchmark and not the gameplay...if you can't set a "follow"
scenario like this, then just pick some parts from your favorite maps during
real-world gameplay and go in and measure perf. with old and new
drivers...in a real-world game scenario, is there real-world benefit?

...in a nutshell, would sure like to see someone looking hard at these
drivers and poking into whether they are just tuned for Q3 benchmarks or
whether they actually buy the user anything in terms of real-world game
performance...

Scott

Four months later, 3dfx got swallowed by NVIDIA.

Just thought you guys may like to know how early I know about this kind of stuff ;) :) Later on, he asked me to record a new demo, to replace my Quaver demo.
 
Rev: I'm gonna sound insane saying this, but...
What about trying what Scott said today? I don't think anyone ever did it, and I think it might give us a most excellent reality check.

If it was true, then we'd begin to really have a case against nVidia - enough evidence to either compeltely tarnish their reputation ( which I'd prefer not, because I'm still looking foward to the NV40, got good reasons to believe it'll be on time and much better than the NV3x ) , or make them stop all this darn crap.

I've got an Hercules GF2 GTS in one of my old PCs - don't have Quake 3 though. Is it possible to use that timedemo with a demo?


Uttar
 
Hellbinder said:
Ok I am having some problems with some of the other numbers in this review like these

http://firingsquad.gamers.com/hardware/msi_geforce_fx5900-td128_review/page12.asp

I simply do not believe that the Nvidia card is really doing AA+AF on this game. There is just no freking way that the Fx is going to be 39 FPS faster than the 9800pro with AA+AF applied.

I repeat.. NO WAY.

I find a couple other oddities in the numbers as well. Some of it can be accounted for becuase of the Nvidia cards Core Speed advantage. But..

Both the results for Nascar 2003 and IL-S have me scratching my head. These are probably the two games where you have the largest potential draw distances. The rest are FPS games IIRC, which don't always have a large draw distance unless you're in space. Before I claim any nV shenanigans, maybe some B3D brain trust/reasoning could chime as to why this could occur. Could it be possible that nV has better algorithms wrt draw distances than ATi? Could it be a case where less need/use of culling could have a significant effect memory or otherwise? I'm just speculatively rambling, so if it sounds like I don't know what I'm talking about, you're right. :D Cheating isn't out of the question either, but I don't want to go down that route unless someone can provide hard proof.
 
Back
Top