NV3X subpixel precision in 3DMark03

Lets be clear here, Nvidia brought a demo they made to ID, Carmack didn't like it, made another BUT that does tell us that they had a build of Doom 3 that was recent, and they have tweaked their drivers for Doom 3..obviously. These optimizations I'm sure can be done at the driver/game engine level and does not need to be just for a generic time demo.

 

Paul sent me to where it was said that the timedemo wasn't created till the day of the test, so I would have to agree with both of you with regard to NVIDIA trying to use the same method with the Doom3 test. Unfortunately for NVIDIA it still makes me suspect to any future timedemo tests they may run.

Tommy McClain

 

Their shader optimisation is now at least part-automated (they have tools that do the transcoding, although it is still mostly done by hand, AFAIK). Surely it would have been possible to write similar macros to rapidly "optimise" for a benchmark at short notice by recording geometry and software culling, (at least in part)?

I agree that it is unlikely this is what they have done with the Doom 3 benchmark though.

MuFu.

 

Well, they could just record the bmps of all the frames and play them back.

But do you really think they're going to that level?

 

yesh...?

 

Well, futuremark is either creaming in their pants or crapping in their pants at this point.

Whether it was the intention or not, NVIDIA has essentially played jinga with the benchmark world and pulled out the wrong piece. The entire set of benchmarks out there are suspect. Just perusing the forums, even the layman is coming up with ingenous ways to cheat. Demo on a rail, snoop for timedemo start and cache the data to post process to improve the scores the next time (yes, Althornin, in retrospect, I think it would be possible to 'optimize' for a given map/timedemo--even just running the timedemo once), snoop for markers and execute a highly optimized code set to accellerate that useage model of the benchmark, snoop for certain keypresses. All sorts of goodies that would invalidate test scores.

It might just be me being melodramatic but it seems 3dmk3 is dead as a useful tool--even if NVIDIA didn't do what they've been accused of 'on purpose', it still is shown that it can be done which taints the whole data set--both past and future.

But, is it a disaster or an opportunity for FutureMark?

 

How is it a disaster, Beta Members have the ability to examine every frame..there can be no cheating here.
Me being one of the most outspoken people against the benchmark believes this new 3Dmark 03 to be one of the most significant benchmarks to use today.

There is no more gettting away with LOD bias hacks
No more non-whql drivers
Policing of the benchmark (way overdue)
No more 'splash screens'
No more '16 bit hackery'

 

Really? You feel that secure? I don't.

 

Doh! How will that stop it exactly?

 

Nothing is 'fool proof', otherwise I wouldn't be downloading Virus Definitions every week.

This is a start though...

Easily, like last night..expose it publicly...frame by frame capture features...better than 3Dmark 2001 for sure.

 

I don't see it affecting 3DMark negatively at all. There's nothing nVidia can do to kill 3DMark as a popular benchmark--they might as well knock their heads against a wall... Many people like it--and as long as that's the case nVidia's behavior will be largely viewed as nothing but sour grapes laced with a generous dollop of hypocrisy.

The only ones hurt in the long run through this will be nVidia. I don't see much chance that nVidia will be able to abolish popular software merely by rigging its drivers to cheat it. nVidia might abolish nVidia that way, though... Perhaps nVidia is what has been tainted here? Just a thought...

 

A few quick quotes:

Anandtech didn't use 3DMark2K3 or 2K1 in their 5900U review.

 

Funny, they use codecreatures though. Which is not a game but a benchmark. It is also succeptible (spelling?) to the same type of cheat. Kind of hypocritical. At least with 3dmark03 the beta members have an opportunity to check for things like this cheat. At least with 3dmark they have the beta program where they get input from many hardware and software companies, as well as review sites.

 

The way to solve this problem is to randomize the demos, but perform a high number of trials. You don't really need perfect repeatability (if you are not comparing screenshots), you just need to run enough trials to reduce the chances that one video card got a "bad" test or that the cheaters can know ahead of time what they are going to render.

In the field of crypography, this is called cut and choose. Have the 3D benchmark issue a large set of random jobs for workload. The 3D driver then have to work on all of them, without knowing which of them will be used for the final benchmark. Then, the 3D benchmark app selects a random subset of the finished jobs as a basis for the benchmark numbers.

You can increase the size of the workload and subset to reduce the probability of foreknowledge, or the probability of getting too many "unfair" jobs.


Even timedemos are not safe, because they are prerecorded. Ideally, the benchmark should be a bot-like AI player seeded randomly playing the game as a normal person would.

 

this thread just got carried away to 3dmark2003 cheating but I think the original problem Ante P saw is worthwhile analyzing.

 

I will digress a little bit but it is still fairly related to what you said re game engine "optimizations" and timedemo "cheating/optimizations".

Is it possible that if Carmack allows you to specify "r_renderer ARB2" (not sure if this is an actual existing variable, but you should get the drift, which is for a user to specify rendering path) and you actually do so, that ATi or NVIDIA would have a tool (a detection mechanism, so to speak) that if such a path will not result in the fastest path possible for either (because they already know it to be so... both IHVs definitely should have the D3 source codes which they must continuously be liasing with Carmack for any significant changes/updates), that they could "turn" the specified path into a "r_renderer R200" or "r_renderer NV30" path (respectively so) by extracting the code, recode it and put in the changed code?

I think it is possible. I think it is actually being done although I don't know if both ATi and NVIDIA are doing it or if it is only either one.

Would any benchmarker know of such an "optimization"? How willing is a benchmarker to study -- I mean REALLY study -- any pptential IQ differences? Do they know how a ARB2 path should look like? Would they also know how a R200 or NV30 path look like? Would they know what possible differences amongst the paths would be without being pointed out to them?

How wrong would it be? For what reasons?

 

I suppose then you're at the mercy of the developer taking their time to verify this isn't happening.

Just like multiple random patterns may work to dispel the culling cheats we're now debating, this means more work for everyone.

(I'd like to hear another "world-wise" poster tell me that this doesn't matter, because everyone cheats, that's how the world works, blah blah blah. I haven't seen this here yet, but I have at both Anand and even Ars.)

 

Far be it for me to put words in Futremarks mouth, it wouldn't at all surprise me to find that there will at some point be a patch to 3DM03 that makes the free cam mode available (although I'm not entirely sure that will happen yet). I've also suggested putting a coordinate system on the display for ease of repeatability.

 

Oh sure, you can change variables on the fly. It's really only an extension upon the whole "auto detect" settings in many modern games. The key point is that you'd then have to change the variable back at the end of the benchmark, since (using your example) you could simply type "r_renderer" and it would give you the currently set state - "r_renderer NV30" for example.

I wouldn't call that cheating per se, although i guess it is deceiving the user. They're using a renderer which the developer has legitimately coded in, which is fine - They're just being dishonest by not telling the user, or letting the user force the renderer he/she wants. It's really no different to lowering the quality settings to "Performance" or whatever in the drivers - It's not technically wrong.

With Doom3 it would be very possible to get away with it, unless you were actively looking for the "cheat". This is possible due to the fact that the ARB2 and NV30 renderers look almost identical, and any other minor differences could be put down to driver differences (since no two cards look absolutely identical). Again though, as i've said before, i find it difficult to believe it's been done here. When the game is released though... well, that's a different story.

 

I'd call it deceiving the end user at their expense, a form of cheating. It would be technically wrong, as the user/game is asking for X, and receiving Y. It doesn't matter that Y looks 95% as good as X, just like Performance AF may look 95% as good as Quality to some users--the user asked for X, and the card should deliver.