NV3X subpixel precision in 3DMark03

Discussion in 'Architecture and Products' started by Ante P, May 14, 2003.

  1. Doomtrooper

    Veteran

    Joined:
    Feb 6, 2002
    Messages:
    3,328
    Likes Received:
    0
    Location:
    Ontario, Canada
    Lets be clear here, Nvidia brought a demo they made to ID, Carmack didn't like it, made another BUT that does tell us that they had a build of Doom 3 that was recent, and they have tweaked their drivers for Doom 3..obviously. These optimizations I'm sure can be done at the driver/game engine level and does not need to be just for a generic time demo.
     
  2. AzBat

    AzBat Agent of the Bat
    Legend

    Joined:
    Apr 1, 2002
    Messages:
    7,749
    Likes Received:
    4,847
    Location:
    Alma, AR
    Paul sent me to where it was said that the timedemo wasn't created till the day of the test, so I would have to agree with both of you with regard to NVIDIA trying to use the same method with the Doom3 test. Unfortunately for NVIDIA it still makes me suspect to any future timedemo tests they may run.

    Tommy McClain
     
  3. MuFu

    MuFu Chief Spastic Baboon
    Veteran

    Joined:
    Jun 12, 2002
    Messages:
    2,258
    Likes Received:
    51
    Location:
    Location, Location with Kirstie Allsopp
    Their shader optimisation is now at least part-automated (they have tools that do the transcoding, although it is still mostly done by hand, AFAIK). Surely it would have been possible to write similar macros to rapidly "optimise" for a benchmark at short notice by recording geometry and software culling, (at least in part)?

    I agree that it is unlikely this is what they have done with the Doom 3 benchmark though.

    MuFu.
     
  4. RussSchultz

    RussSchultz Professional Malcontent
    Veteran

    Joined:
    Feb 6, 2002
    Messages:
    2,855
    Likes Received:
    55
    Location:
    HTTP 404
    Well, they could just record the bmps of all the frames and play them back.

    But do you really think they're going to that level?
     
  5. Tahir2

    Veteran

    Joined:
    Feb 7, 2002
    Messages:
    2,978
    Likes Received:
    86
    Location:
    Earth
    yesh...? ;)
     
  6. RussSchultz

    RussSchultz Professional Malcontent
    Veteran

    Joined:
    Feb 6, 2002
    Messages:
    2,855
    Likes Received:
    55
    Location:
    HTTP 404
    Well, futuremark is either creaming in their pants or crapping in their pants at this point.

    Whether it was the intention or not, NVIDIA has essentially played jinga with the benchmark world and pulled out the wrong piece. The entire set of benchmarks out there are suspect. Just perusing the forums, even the layman is coming up with ingenous ways to cheat. Demo on a rail, snoop for timedemo start and cache the data to post process to improve the scores the next time (yes, Althornin, in retrospect, I think it would be possible to 'optimize' for a given map/timedemo--even just running the timedemo once), snoop for markers and execute a highly optimized code set to accellerate that useage model of the benchmark, snoop for certain keypresses. All sorts of goodies that would invalidate test scores.

    It might just be me being melodramatic but it seems 3dmk3 is dead as a useful tool--even if NVIDIA didn't do what they've been accused of 'on purpose', it still is shown that it can be done which taints the whole data set--both past and future.

    But, is it a disaster or an opportunity for FutureMark?
     
  7. Doomtrooper

    Veteran

    Joined:
    Feb 6, 2002
    Messages:
    3,328
    Likes Received:
    0
    Location:
    Ontario, Canada
    How is it a disaster, Beta Members have the ability to examine every frame..there can be no cheating here.
    Me being one of the most outspoken people against the benchmark believes this new 3Dmark 03 to be one of the most significant benchmarks to use today.

    There is no more gettting away with LOD bias hacks
    No more non-whql drivers
    Policing of the benchmark (way overdue)
    No more 'splash screens'
    No more '16 bit hackery'
     
  8. RussSchultz

    RussSchultz Professional Malcontent
    Veteran

    Joined:
    Feb 6, 2002
    Messages:
    2,855
    Likes Received:
    55
    Location:
    HTTP 404
    Really? You feel that secure? I don't.
     
  9. BoardBonobo

    BoardBonobo My hat is white(ish)!
    Veteran

    Joined:
    May 30, 2002
    Messages:
    3,605
    Likes Received:
    541
    Location:
    SurfMonkey's Cluster...
    Doh! How will that stop it exactly?
     
  10. Doomtrooper

    Veteran

    Joined:
    Feb 6, 2002
    Messages:
    3,328
    Likes Received:
    0
    Location:
    Ontario, Canada
    Nothing is 'fool proof', otherwise I wouldn't be downloading Virus Definitions every week.

    This is a start though...

    Easily, like last night..expose it publicly...frame by frame capture features...better than 3Dmark 2001 for sure.
     
  11. WaltC

    Veteran

    Joined:
    Jul 22, 2002
    Messages:
    2,710
    Likes Received:
    8
    Location:
    BelleVue Sanatorium, Billary, NY. Patient privile
    I don't see it affecting 3DMark negatively at all. There's nothing nVidia can do to kill 3DMark as a popular benchmark--they might as well knock their heads against a wall...;) Many people like it--and as long as that's the case nVidia's behavior will be largely viewed as nothing but sour grapes laced with a generous dollop of hypocrisy.

    The only ones hurt in the long run through this will be nVidia. I don't see much chance that nVidia will be able to abolish popular software merely by rigging its drivers to cheat it. nVidia might abolish nVidia that way, though...;) Perhaps nVidia is what has been tainted here? Just a thought...
     
  12. DadUM

    Newcomer

    Joined:
    Oct 11, 2002
    Messages:
    55
    Likes Received:
    0
    A few quick quotes:

    Anandtech didn't use 3DMark2K3 or 2K1 in their 5900U review.
     
  13. jjayb

    Regular

    Joined:
    Feb 13, 2002
    Messages:
    358
    Likes Received:
    1
    Funny, they use codecreatures though. Which is not a game but a benchmark. It is also succeptible (spelling?) to the same type of cheat. Kind of hypocritical. At least with 3dmark03 the beta members have an opportunity to check for things like this cheat. At least with 3dmark they have the beta program where they get input from many hardware and software companies, as well as review sites.
     
  14. DemoCoder

    Veteran

    Joined:
    Feb 9, 2002
    Messages:
    4,733
    Likes Received:
    81
    Location:
    California
    The way to solve this problem is to randomize the demos, but perform a high number of trials. You don't really need perfect repeatability (if you are not comparing screenshots), you just need to run enough trials to reduce the chances that one video card got a "bad" test or that the cheaters can know ahead of time what they are going to render.

    In the field of crypography, this is called cut and choose. Have the 3D benchmark issue a large set of random jobs for workload. The 3D driver then have to work on all of them, without knowing which of them will be used for the final benchmark. Then, the 3D benchmark app selects a random subset of the finished jobs as a basis for the benchmark numbers.

    You can increase the size of the workload and subset to reduce the probability of foreknowledge, or the probability of getting too many "unfair" jobs.


    Even timedemos are not safe, because they are prerecorded. Ideally, the benchmark should be a bot-like AI player seeded randomly playing the game as a normal person would.
     
  15. dream caster

    Newcomer

    Joined:
    Jan 4, 2003
    Messages:
    40
    Likes Received:
    0
    Location:
    Chile
    this thread just got carried away to 3dmark2003 cheating but I think the original problem Ante P saw is worthwhile analyzing.
     
  16. Reverend

    Banned

    Joined:
    Jan 31, 2002
    Messages:
    3,266
    Likes Received:
    24
    I will digress a little bit but it is still fairly related to what you said re game engine "optimizations" and timedemo "cheating/optimizations".

    Is it possible that if Carmack allows you to specify "r_renderer ARB2" (not sure if this is an actual existing variable, but you should get the drift, which is for a user to specify rendering path) and you actually do so, that ATi or NVIDIA would have a tool (a detection mechanism, so to speak) that if such a path will not result in the fastest path possible for either (because they already know it to be so... both IHVs definitely should have the D3 source codes which they must continuously be liasing with Carmack for any significant changes/updates), that they could "turn" the specified path into a "r_renderer R200" or "r_renderer NV30" path (respectively so) by extracting the code, recode it and put in the changed code?

    I think it is possible. I think it is actually being done although I don't know if both ATi and NVIDIA are doing it or if it is only either one.

    Would any benchmarker know of such an "optimization"? How willing is a benchmarker to study -- I mean REALLY study -- any pptential IQ differences? Do they know how a ARB2 path should look like? Would they also know how a R200 or NV30 path look like? Would they know what possible differences amongst the paths would be without being pointed out to them?

    How wrong would it be? For what reasons?
     
  17. Pete

    Pete Moderate Nuisance
    Moderator Legend

    Joined:
    Feb 7, 2002
    Messages:
    5,777
    Likes Received:
    1,814
    I suppose then you're at the mercy of the developer taking their time to verify this isn't happening.

    Just like multiple random patterns may work to dispel the culling cheats we're now debating, this means more work for everyone.

    (I'd like to hear another "world-wise" poster tell me that this doesn't matter, because everyone cheats, that's how the world works, blah blah blah. :roll: I haven't seen this here yet, but I have at both Anand and even Ars.)
     
  18. Dave Baumann

    Dave Baumann Gamerscore Wh...
    Moderator Legend

    Joined:
    Jan 29, 2002
    Messages:
    14,090
    Likes Received:
    694
    Location:
    O Canada!
    Far be it for me to put words in Futremarks mouth, it wouldn't at all surprise me to find that there will at some point be a patch to 3DM03 that makes the free cam mode available (although I'm not entirely sure that will happen yet). I've also suggested putting a coordinate system on the display for ease of repeatability.
     
  19. PaulS

    Regular

    Joined:
    May 12, 2003
    Messages:
    481
    Likes Received:
    1
    Location:
    UK
    Oh sure, you can change variables on the fly. It's really only an extension upon the whole "auto detect" settings in many modern games. The key point is that you'd then have to change the variable back at the end of the benchmark, since (using your example) you could simply type "r_renderer" and it would give you the currently set state - "r_renderer NV30" for example.

    I wouldn't call that cheating per se, although i guess it is deceiving the user. They're using a renderer which the developer has legitimately coded in, which is fine - They're just being dishonest by not telling the user, or letting the user force the renderer he/she wants. It's really no different to lowering the quality settings to "Performance" or whatever in the drivers - It's not technically wrong.

    With Doom3 it would be very possible to get away with it, unless you were actively looking for the "cheat". This is possible due to the fact that the ARB2 and NV30 renderers look almost identical, and any other minor differences could be put down to driver differences (since no two cards look absolutely identical). Again though, as i've said before, i find it difficult to believe it's been done here. When the game is released though... well, that's a different story.
     
  20. Pete

    Pete Moderate Nuisance
    Moderator Legend

    Joined:
    Feb 7, 2002
    Messages:
    5,777
    Likes Received:
    1,814
    I'd call it deceiving the end user at their expense, a form of cheating. It would be technically wrong, as the user/game is asking for X, and receiving Y. It doesn't matter that Y looks 95% as good as X, just like Performance AF may look 95% as good as Quality to some users--the user asked for X, and the card should deliver.
     
Loading...

Share This Page

  • About Us

    Beyond3D has been around for over a decade and prides itself on being the best place on the web for in-depth, technically-driven discussion and analysis of 3D graphics hardware. If you love pixels and transistors, you've come to the right place!

    Beyond3D is proudly published by GPU Tools Ltd.
Loading...