Use of Custom Demo's In Reviews

Discussion in 'Graphics and Semiconductor Industry' started by Dave Baumann, Jun 16, 2003.

  1. Dave Baumann

    Dave Baumann Gamerscore Wh...
    Moderator Legend

    Joined:
    Jan 29, 2002
    Messages:
    14,079
    Likes Received:
    648
    Location:
    O Canada!
    We've talked about using custom demos in reviews since this 3DMark stuff blew up. Brandon Bell over at FS has done just that with both Quake 3 and SS:SE - normally these reside in the playground of NVIDIA, but look at the order of the scores when a non-standard benchmark is used:

    http://firingsquad.gamers.com/hardware/msi_geforce_fx5900-td128_review/page8.asp

    http://firingsquad.gamers.com/hardware/msi_geforce_fx5900-td128_review/page9.asp

    Not exactly the orders we'd normally expect to see for these two benchmarks.
     
  2. breez

    Regular

    Joined:
    Aug 13, 2002
    Messages:
    270
    Likes Received:
    1
    Location:
    Finland
    Well that is interesting 8)
     
  3. Natoma

    Veteran

    Joined:
    Feb 6, 2002
    Messages:
    1,913
    Likes Received:
    84
    It's funny. I remember bringing up the possibility of Nvidia doing the same kind of driver "optimizations" for games that have timedemo systems, and being laughed at by some people for such a preposterous notion.

    Well, my purchase decision has been made. Definitely voting with my wallet on this one.
     
  4. LeStoffer

    Veteran

    Joined:
    Feb 6, 2002
    Messages:
    1,253
    Likes Received:
    13
    Location:
    Land of the 25% VAT
  5. sonix666

    Regular

    Joined:
    Mar 31, 2003
    Messages:
    595
    Likes Received:
    3
    Dave,

    besides the custom benchmarks, the following differences there are between this review and the last one they did:

    - Serious Sam SE is run in quality mode instead of normal mode.
    - Quake III is run with the latest point release instead of the version they normally use for testing.
    - Quake III is run at 4x FSAA and 16x anisotropic filtering while their previous review didn't use those settings at all (as far as I can see).

    So, the swapped places may actually be a result of the different settings than the custom demo. The Radeon 9700 against the Geforce FX always way much stronger in high quality settings in games.

    Besides the differences there can be many differences between the standard demos and the ones they recorded, so different areas of the GPU might be differently stressed.
     
  6. K.I.L.E.R

    K.I.L.E.R Retarded moron
    Veteran

    Joined:
    Jun 17, 2002
    Messages:
    2,952
    Likes Received:
    50
    Location:
    Australia, Melbourne
    Your idea wasn't shot down by many as I have also stated that nVIDIA WAS optimising for timedemos. I also had thought it was pretty obvious and that everyone had already known. Apparently not.
     
  7. indio

    Newcomer

    Joined:
    Feb 1, 2003
    Messages:
    221
    Likes Received:
    0
    Location:
    Berkshire County , Ma USA
    I think this became obvious atleast to me when I saw the review over at Nvnews. Mike C. used Fraps and non-standard games and the 5900U scores weren't very impressive. Then if you look at other non standard games from other reviews like the NOLF2 getting beat by a 9700 pro it becomes quite obvious.
     
  8. Tokelil

    Regular

    Joined:
    Mar 27, 2002
    Messages:
    329
    Likes Received:
    0
    Location:
    Denmark
    Didn't know that Nvidia supported 16x AF... :shock: :wink:

    Anyway, I think you have a point... We need more tests to be sure that the scores are due to Nvidia "optimizing"...
     
  9. K.I.L.E.R

    K.I.L.E.R Retarded moron
    Veteran

    Joined:
    Jun 17, 2002
    Messages:
    2,952
    Likes Received:
    50
    Location:
    Australia, Melbourne
    Do we have reason to believe they are not optimising?
    With the recent Futuremark events I am sure many people distrust nVIDIA to the point where any high scores with their hardware automatically results in them "optimising" and I for one am not surprised if that is the case 100% of the time.
     
  10. jpeter

    Newcomer

    Joined:
    May 24, 2002
    Messages:
    30
    Likes Received:
    1
    I just put a link to this thread in nvnews and automatically get erase.
    Why? Have just been moved to K.I.L.E.R thread. (http://www.nvnews.net/vbulletin/showthread.php?s=&threadid=13456)

    This is what I wrote:

    Interesting numbers for quake3 and SS:SE have been shown
    by Brandon Bell at FS.



    Writen by DaveBaumann in beyond3d
    http://www.beyond3d.com/forum/viewtopic.php?t=6459

    quote:
    We've talked about using custom demos in reviews since this 3DMark stuff blew up. Brandon Bell over at FS has done just that with both Quake 3 and SS:SE - normally these reside in the playground of NVIDIA, but look at the order of the scores when a non-standard benchmark is used: http://firingsquad.gamers.com/hardw...eview/page8.asp http://firingsquad.gamers.com/hardw...eview/page9.asp Not exactly the orders we'd normally expect to see for these two benchmarks.
     
  11. digitalwanderer

    digitalwanderer Dangerously Mirthful
    Legend

    Joined:
    Feb 19, 2002
    Messages:
    17,267
    Likes Received:
    1,783
    Location:
    Winfield, IN USA
    Ouchers for nVidia.

    I'm not really familar with firingsquad or the gentleman who did the review, but the numbers look pretty consistant to me and like whoever did it knew what they were doing....could someone vouch for the place?
     
  12. K.I.L.E.R

    K.I.L.E.R Retarded moron
    Veteran

    Joined:
    Jun 17, 2002
    Messages:
    2,952
    Likes Received:
    50
    Location:
    Australia, Melbourne
    Read my post that I have made. :)
    It was near yours.


     
  13. Sabastian

    Regular

    Joined:
    Feb 6, 2002
    Messages:
    991
    Likes Received:
    1
    Location:
    Canada
    jpeter: The reason is that nvidia really took some lumps with their cheating on FutureMark and now it is being revealed that they are doing much the same "optimizations" in game benchmarks. The deletion of that post on the nvidia fan site "nvnews" sounds like damage control. If word spreads that nvidia is doing benchmark "optimizations" for games as well then it may result in the same sorts of accusations that FutureMark implied with their test, and that would be a very bad outcome for nvidia. Anyhow, not that you didn't realize why your thread was deleted so quickly, that is the reason, IMO of course.
     
  14. Hellbinder

    Banned

    Joined:
    Feb 8, 2002
    Messages:
    1,444
    Likes Received:
    12
    Ok I am having some problems with some of the other numbers in this review like these

    http://firingsquad.gamers.com/hardware/msi_geforce_fx5900-td128_review/page12.asp

    I simply do not believe that the Nvidia card is really doing AA+AF on this game. There is just no freking way that the Fx is going to be 39 FPS faster than the 9800pro with AA+AF applied.

    I repeat.. NO WAY.

    I find a couple other oddities in the numbers as well. Some of it can be accounted for becuase of the Nvidia cards Core Speed advantage. But..
     
  15. K.I.L.E.R

    K.I.L.E.R Retarded moron
    Veteran

    Joined:
    Jun 17, 2002
    Messages:
    2,952
    Likes Received:
    50
    Location:
    Australia, Melbourne
    His post was merged with mine as it was about the exact same thing. :)
     
  16. Sabastian

    Regular

    Joined:
    Feb 6, 2002
    Messages:
    991
    Likes Received:
    1
    Location:
    Canada
    heh, shouldn't jump to conclusions so quickly. I think I will go lay down now.
    :oops: :oops: :oops: :oops:
     
  17. Reverend

    Banned

    Joined:
    Jan 31, 2002
    Messages:
    3,266
    Likes Received:
    24
    FYI : Scott Sellers, when 3dfx was still around, wrote Kristof, Dave Barron and me the following, just after the release of that infamous 6.16 drivers back in Aug'2000 (GF2 GTS days) :

    Four months later, 3dfx got swallowed by NVIDIA.

    Just thought you guys may like to know how early I know about this kind of stuff ;) :) Later on, he asked me to record a new demo, to replace my Quaver demo.
     
  18. K.I.L.E.R

    K.I.L.E.R Retarded moron
    Veteran

    Joined:
    Jun 17, 2002
    Messages:
    2,952
    Likes Received:
    50
    Location:
    Australia, Melbourne
    I would like to see an auto-timedemo generator in games.
     
  19. Arun

    Arun Unknown.
    Moderator Legend Veteran

    Joined:
    Aug 28, 2002
    Messages:
    5,023
    Likes Received:
    299
    Location:
    UK
    Rev: I'm gonna sound insane saying this, but...
    What about trying what Scott said today? I don't think anyone ever did it, and I think it might give us a most excellent reality check.

    If it was true, then we'd begin to really have a case against nVidia - enough evidence to either compeltely tarnish their reputation ( which I'd prefer not, because I'm still looking foward to the NV40, got good reasons to believe it'll be on time and much better than the NV3x ) , or make them stop all this darn crap.

    I've got an Hercules GF2 GTS in one of my old PCs - don't have Quake 3 though. Is it possible to use that timedemo with a demo?


    Uttar
     
  20. SmuvMoney

    Newcomer

    Joined:
    May 24, 2003
    Messages:
    67
    Likes Received:
    0
    Location:
    Chicago, IL
    Both the results for Nascar 2003 and IL-S have me scratching my head. These are probably the two games where you have the largest potential draw distances. The rest are FPS games IIRC, which don't always have a large draw distance unless you're in space. Before I claim any nV shenanigans, maybe some B3D brain trust/reasoning could chime as to why this could occur. Could it be possible that nV has better algorithms wrt draw distances than ATi? Could it be a case where less need/use of culling could have a significant effect memory or otherwise? I'm just speculatively rambling, so if it sounds like I don't know what I'm talking about, you're right. :D Cheating isn't out of the question either, but I don't want to go down that route unless someone can provide hard proof.
     
Loading...

Share This Page

  • About Us

    Beyond3D has been around for over a decade and prides itself on being the best place on the web for in-depth, technically-driven discussion and analysis of 3D graphics hardware. If you love pixels and transistors, you've come to the right place!

    Beyond3D is proudly published by GPU Tools Ltd.
Loading...