[Natoma]

Uhm, I don't know if this has been brought up or not. But given that timedemos are "on the rails" so to speak as are the 3DMark tests, is it not possible that Nvidia and other companies could do these same sort of hacks to improve performance that would not show up in real world gaming?

I mean, no one has a Quake3 SDK to be able to take the timedemos off the rails and test whether or not there's any funny business going on right?

I know 3DMark is synthetic, but so is any benchmark that you can't actually play. Any timedemo is synthetic, and thus would probably be subject to this same type of manipulation. I don't know if I'd put it past them.

Thoughts?

[Colourless]

Theoretically, detecting a Quake 3 timedemo run would be reletively easy. The loading screen is always the same when loading the demo.

As for going of the rails. If there are any quake 3 demo editors, they could be used to modify the path in the demo.

[Pete]

Originally Posted by Natoma

I mean, no one has a Quake3 SDK to be able to take the timedemos off the rails and test whether or not there's any funny business going on right?

Yes, we all do: Quake 3, the game (and a copy of FRAPS). Voila, you're off the rails (with automatic "timelapse" framerate recording). Or just create your own custom demo (every time you benchmark, if you end up releasing it with the review, or once, if you keep it private).

You could compare a timedemo of walking through a level to a FRAPS run of the same thing, and that should provide roughly equal settings for you to compare timedemo to actual numbers.

As has been discussed in the many threads pertaining to ET's article, all timedemos are not automatically invalid, but they are automatically in question. Reviewers will have to work that much harder to verify that the benchmarks they run are indicative of real performance.

[Dio]

Originally Posted by Colourless

Theoretically, detecting a Quake 3 timedemo run would be reletively easy.

There's no theoretically about it.

[dksuiko]

I think custom timedemos are better than the standard 'demo1's and 'demo2's as tools for a review (with the same demo up for download). The only drawback I see for going this route would be the inability to compare each site's benchmarks. And I don't think that's too much of a bad thing. We'd actually get different tests from more reviews, putting the videocard is more situations than a 'demo1' can ever do.

[duncan36]

Is it even as simple as being on the rails being the key determinant for cheating? The rabbit hole keeps getting deeper.

[Xspringe]

Originally Posted by duncan36

Is it even as simple as being on the rails being the key determinant for cheating? The rabbit hole keeps getting deeper.

Soon we'll be having something like punkbuster for benchmarks...

[Reverend]

The following is identical to what I posted here :

We make our own custom demos without releasing it to the public nor even say what level that demo is based on. There is no doubt now that recorded demos are extremely susceptible to what NVIDIA has done with 3DMark03's various tests and if a recorded game demo is available to the public, it will be just as susceptible. Even if we don't make the demo available to the public, I will be forever paranoid/pessimistic/distrustful and assume that if we even mention what level such a recorded demo was based on, IHVs will simply go and mess with the entire level.

Like I said in one of my past posts, the trust is lost and I am not certain how it may be regained.

[Reverend]

I would have to add, of course, that I don't really know how acceptable my above suggestion is to the general public.

[Dave Baumann]

I'm already highly suspicious of our Splinter Cell benchmark.

[Archaeolept]

Originally Posted by DaveBaumann

I'm already highly suspicious of our Splinter Cell benchmark.

The Radeon bench's? Those are the only SC ones I can see.

[Reverend]

Dave is referring to our own custom Splinter Cell demo, which you can find here.

In the past few days, Brent of HardOCP brought to light (to Dave) something he noticed while running this demo of ours (and I do mean our demo, not actually playing the game in that level) on a GFFX 5900Ultra. This demo has an ocean in its scene (a BIG ocean) and the entire ocean surface has two ps_1_1 shaders to effect, well, an ocean's surface.

I know exactly what those two ps_1_1 shaders codes are (unfortunately I can't provide them -- Ubisoft doesn't want me to) and those codes mean that "waves/ripples" should be rendered over the entire ocean's surface. Brent noticed that there are a number of spots where there are no "waves/ripples" on his 5900Ultra using 44.03s. Dave and I have confirmed that this "anomaly" happens only on 5600 and up but not on a 5200. I can confirm that both the demo and in actual gameplay in that level do not show up any such bug on a 5200 and Dave will investigate whether the bug exists in both the demo and in the gameplay level on a 5600 and above.

Sorry if you're asking for screenshots to depict what my attempt above to describe it -- I don't think it is right to provide these and fuel speculations until we get to the bottom of what's happening.

Dany Lepage (Ubisoft) has provided a very rough comment and I don't want to quote him in its entirety except that he thinks it is "definitely a driver bug". The thing is that we're trying to figure out why this "bug" exists on NV30/31/35 and not on NV34 even with NVIDIA's Unified Driver Architecture.

As for why Dave is highly suspicious, I have found out (and told Dave) that NVIDIA has a personnel focussed solely on SplinterCell benchmarking.

More, if there is more to be revealed, when we find more concrete information. For now, just take all of the above as no more than our providing our findings as we see them. I'd like to think it's a driver bug obviously but we read the above paragraph it's natural to become suspicious given the recent issues.

[Pete]

Well, that's disturbing. I was hoping my fears of this changing benchmarks as we know them were overblown, but it looks like reviewers (and developers) may have their work cut out for them. (You can apply clipping planes to that work, but it would be a disservice to your readers. :? )

[demalion]

That sounds like a universal change, not a demo specific one. It doesn't look like there will be any easy substitute for vigilant image quality analysis. Also, for SC, things like shadow detail level seem prone to driver level customization, since I know that was a major performance factor for my card when I ran the demo.

[Cuervo]

What about nVidia supplying all the computers and demo for Doom III? We surely can't trust those numbers now. No matter who did the benchmarking on them it would seem they are also in question. Hell, anything nVidia does from this point on will be questioned. How could they think no one would fined out about it.

-Cuervo

[Lezmaka]

Originally Posted by Cuervo

What about nVidia supplying all the computers and demo for Doom III? We surely can't trust those numbers now. No matter who did the benchmarking on them it would seem they are also in question. Hell, anything nVidia does from this point on will be questioned. How could they think no one would fined out about it.

-Cuervo

For one of the sites, the computer didn't arrive, so the site was allowed to build their own system.

[ET]

Originally Posted by Reverend

The thing is that we're trying to figure out why this "bug" exists on NV30/31/35 and not on NV34 even with NVIDIA's Unified Driver Architecture.

Unified Driver Architecture doesn't mean that the hardware is the same. I've encountered different behaviour of a GeForce3 to other GeForce cards before, because it forces clipping in all situations. Nothing the driver can do about that. So it's not unthinkable that the NV34 is different enough to not show this problem, while the other chips share the same hardware issue.

[bloodbob]

Natoma of posted similar comments on other forums I totally agree

Dave is referring to our own custom Splinter Cell demo, which you can find here.

In the past few days, Brent of HardOCP brought to light (to Dave) something he noticed while running this demo of ours (and I do mean our demo, not actually playing the game in that level) on a GFFX 5900Ultra. This demo has an ocean in its scene (a BIG ocean) and the entire ocean surface has two ps_1_1 shaders to effect, well, an ocean's surface.

I know exactly what those two ps_1_1 shaders codes are (unfortunately I can't provide them -- Ubisoft doesn't want me to) and those codes mean that "waves/ripples" should be rendered over the entire ocean's surface. Brent noticed that there are a number of spots where there are no "waves/ripples" on his 5900Ultra using 44.03s. Dave and I have confirmed that this "anomaly" happens only on 5600 and up but not on a 5200. I can confirm that both the demo and in actual gameplay in that level do not show up any such bug on a 5200 and Dave will investigate whether the bug exists in both the demo and in the gameplay level on a 5600 and above.

Sorry if you're asking for screenshots to depict what my attempt above to describe it -- I don't think it is right to provide these and fuel speculations until we get to the bottom of what's happening.

Dany Lepage (Ubisoft) has provided a very rough comment and I don't want to quote him in its entirety except that he thinks it is "definitely a driver bug". The thing is that we're trying to figure out why this "bug" exists on NV30/31/35 and not on NV34 even with NVIDIA's Unified Driver Architecture.

As for why Dave is highly suspicious, I have found out (and told Dave) that NVIDIA has a personnel focussed solely on SplinterCell benchmarking.

More, if there is more to be revealed, when we find more concrete information. For now, just take all of the above as no more than our providing our findings as we see them. I'd like to think it's a driver bug obviously but we read the above paragraph it's natural to become suspicious given the recent issues.

Few things here okay now I'm gonna be an *nasty word incase they ain't allowed* and say that kyle has been biased on this issue generally I like what he does but this time no. If kyle was to record a custom demo(s) for hardocp and keep them private how is the community supposed to know his scores are valid. Lets propose you have a game with levels that only partially use PS now if the demo record was to run around the level and view the only the parts without the pixel shaders and save it is that going to be even? I think not there are obivusly ways to make private demos strongly biased. Now even with private demos you can always just drop the IQ when timedemo are detected as stated by a number of people so its still not the answer. I'm sure publishers of games don't want the develpoers to be running around breaking cheating code in drivers they are wanting fixes/new games so its hard for game developers to stop this.

Brent has seemed to be completely opened minded about the whole thing I find its very sad that kyle has not said one word since the news has come out on his forum.

[YeuEmMaiMai]

so lets just get down to the $500 question shall we?


who in their right mind would buy a 5900 ultra over a radeon 9800 pro (knowing that from what has been said) seeings that their drivers are


1. extremely buggy and therefore cannot correctly render features in your favorite games not giving you the IQ that you would EXPECT AND PAID FOR in that $500 card.

or

2. hacked beyond belief so that they can compete with the competition not giving you the IQ that you would EXPECT AND PAID FOR in that $500 card.

Funny how all the diehard nVidiots state "ATi drivers are crap and full of bugs" all the while it appears that nVidia drivers are far more "bug" ridden than anyone wants to admitt...

Personally this is the best part of all....when I play my games I get the same frame rates on the maps that the demos are recorded on for the most part. You will never get the exaxt same frames as your CPU is busy doing other things such as AI.........where in a time demo that part does NOT have to be done.

[LeStoffer]

Originally Posted by Reverend

As for why Dave is highly suspicious, I have found out (and told Dave) that NVIDIA has a personnel focussed solely on SplinterCell benchmarking.

Originally Posted by Reverend

I'd like to think it's a driver bug obviously but we read the above paragraph it's natural to become suspicious given the recent issues.

So very true, unfortunately. And with high school kids running nVidia's PR department these days I'm sorry to say they probably wont help rebuild any trust anytime soon.

[Reverend]

And with high school kids running nVidia's PR department these days I'm sorry to say they probably wont help rebuild any trust anytime soon.

Whoop! Brian Burke would love this one!

I haven't talked to Brian about all of this. Knowing him, I think he's just torn between being loyal to the company he works for, and being religious. Heh.

[LeStoffer]

Originally Posted by Reverend

I haven't talked to Brian about all of this. Knowing him, I think he's just torn between being loyal to the company he works for, and being religious. Heh.

Yeah, it would be interesting the know what he think about this mess. :P

Anyway Rev, my comment was really regarding the absurdy bad statement the PR department sent out as damage control. Totally unprofessionel high school quality in my view and I'm sure that others in the business has been shaking their heads too. Remember the statement was not some marketing BS (where you're allowed to, well, BS some) but the only official reaction from nVidia to the rest of the industry.

[Typedef Enum]

Personally, I think nVidia's PR has done nothing but gone down the tubes since Brian was brought into the mix...

That might seem really odd, given nVidia's reputation when [What the hecks his name? The head PR guy that many people hated?] was fairly prominent...But I swear up and down that they've actually gotten significantly worse since he arrived...and when I say worse, I'm talking like towards_the_end_of_3dfx worse, if you catch my drift...IE save the sinking ship kinda' worse...

[Pete]

Derek Perez? There was a woman, too, though now I can't remember her name. :?

[RussSchultz]

Diane Vanasse

EDIT: finally overcame my lazyness and looked it up