Are all timedemos now invalid because of this 3DMark Fiasco?

Natoma

Veteran
Uhm, I don't know if this has been brought up or not. But given that timedemos are "on the rails" so to speak as are the 3DMark tests, is it not possible that Nvidia and other companies could do these same sort of hacks to improve performance that would not show up in real world gaming?

I mean, no one has a Quake3 SDK to be able to take the timedemos off the rails and test whether or not there's any funny business going on right?

I know 3DMark is synthetic, but so is any benchmark that you can't actually play. Any timedemo is synthetic, and thus would probably be subject to this same type of manipulation. I don't know if I'd put it past them.

Thoughts?
 
Theoretically, detecting a Quake 3 timedemo run would be reletively easy. The loading screen is always the same when loading the demo.

As for going of the rails. If there are any quake 3 demo editors, they could be used to modify the path in the demo.
 
Re: Are all timedemos now invalid because of this 3DMark Fia

Natoma said:
I mean, no one has a Quake3 SDK to be able to take the timedemos off the rails and test whether or not there's any funny business going on right?
Yes, we all do: Quake 3, the game (and a copy of FRAPS). Voila, you're off the rails (with automatic "timelapse" framerate recording). Or just create your own custom demo (every time you benchmark, if you end up releasing it with the review, or once, if you keep it private).

You could compare a timedemo of walking through a level to a FRAPS run of the same thing, and that should provide roughly equal settings for you to compare timedemo to actual numbers.

As has been discussed in the many threads pertaining to ET's article, all timedemos are not automatically invalid, but they are automatically in question. Reviewers will have to work that much harder to verify that the benchmarks they run are indicative of real performance.
 
I think custom timedemos are better than the standard 'demo1's and 'demo2's as tools for a review (with the same demo up for download). The only drawback I see for going this route would be the inability to compare each site's benchmarks. And I don't think that's too much of a bad thing. We'd actually get different tests from more reviews, putting the videocard is more situations than a 'demo1' can ever do.
 
Is it even as simple as being on the rails being the key determinant for cheating? The rabbit hole keeps getting deeper.
 
duncan36 said:
Is it even as simple as being on the rails being the key determinant for cheating? The rabbit hole keeps getting deeper.

Soon we'll be having something like punkbuster for benchmarks... :oops:
 
The following is identical to what I posted here :

We make our own custom demos without releasing it to the public nor even say what level that demo is based on. There is no doubt now that recorded demos are extremely susceptible to what NVIDIA has done with 3DMark03's various tests and if a recorded game demo is available to the public, it will be just as susceptible. Even if we don't make the demo available to the public, I will be forever paranoid/pessimistic/distrustful and assume that if we even mention what level such a recorded demo was based on, IHVs will simply go and mess with the entire level.

Like I said in one of my past posts, the trust is lost and I am not certain how it may be regained.
 
I would have to add, of course, that I don't really know how acceptable my above suggestion is to the general public.
 
Dave is referring to our own custom Splinter Cell demo, which you can find here.

In the past few days, Brent of HardOCP brought to light (to Dave) something he noticed while running this demo of ours (and I do mean our demo, not actually playing the game in that level) on a GFFX 5900Ultra. This demo has an ocean in its scene (a BIG ocean) and the entire ocean surface has two ps_1_1 shaders to effect, well, an ocean's surface.

I know exactly what those two ps_1_1 shaders codes are (unfortunately I can't provide them -- Ubisoft doesn't want me to) and those codes mean that "waves/ripples" should be rendered over the entire ocean's surface. Brent noticed that there are a number of spots where there are no "waves/ripples" on his 5900Ultra using 44.03s. Dave and I have confirmed that this "anomaly" happens only on 5600 and up but not on a 5200. I can confirm that both the demo and in actual gameplay in that level do not show up any such bug on a 5200 and Dave will investigate whether the bug exists in both the demo and in the gameplay level on a 5600 and above.

Sorry if you're asking for screenshots to depict what my attempt above to describe it -- I don't think it is right to provide these and fuel speculations until we get to the bottom of what's happening.

Dany Lepage (Ubisoft) has provided a very rough comment and I don't want to quote him in its entirety except that he thinks it is "definitely a driver bug". The thing is that we're trying to figure out why this "bug" exists on NV30/31/35 and not on NV34 even with NVIDIA's Unified Driver Architecture.

As for why Dave is highly suspicious, I have found out (and told Dave) that NVIDIA has a personnel focussed solely on SplinterCell benchmarking.

More, if there is more to be revealed, when we find more concrete information. For now, just take all of the above as no more than our providing our findings as we see them. I'd like to think it's a driver bug obviously but we read the above paragraph it's natural to become suspicious given the recent issues.
 
Well, that's disturbing. o_O I was hoping my fears of this changing benchmarks as we know them were overblown, but it looks like reviewers (and developers) may have their work cut out for them. (You can apply clipping planes to that work, but it would be a disservice to your readers. ;) :? )
 
That sounds like a universal change, not a demo specific one. It doesn't look like there will be any easy substitute for vigilant image quality analysis. Also, for SC, things like shadow detail level seem prone to driver level customization, since I know that was a major performance factor for my card when I ran the demo.
 
What about nVidia supplying all the computers and demo for Doom III? We surely can't trust those numbers now. No matter who did the benchmarking on them it would seem they are also in question. Hell, anything nVidia does from this point on will be questioned. How could they think no one would fined out about it.

-Cuervo
 
Cuervo said:
What about nVidia supplying all the computers and demo for Doom III? We surely can't trust those numbers now. No matter who did the benchmarking on them it would seem they are also in question. Hell, anything nVidia does from this point on will be questioned. How could they think no one would fined out about it.

-Cuervo

For one of the sites, the computer didn't arrive, so the site was allowed to build their own system.
 
Reverend said:
The thing is that we're trying to figure out why this "bug" exists on NV30/31/35 and not on NV34 even with NVIDIA's Unified Driver Architecture.

Unified Driver Architecture doesn't mean that the hardware is the same. I've encountered different behaviour of a GeForce3 to other GeForce cards before, because it forces clipping in all situations. Nothing the driver can do about that. So it's not unthinkable that the NV34 is different enough to not show this problem, while the other chips share the same hardware issue.
 
Natoma of posted similar comments on other forums I totally agree :)

Dave is referring to our own custom Splinter Cell demo, which you can find here.

In the past few days, Brent of HardOCP brought to light (to Dave) something he noticed while running this demo of ours (and I do mean our demo, not actually playing the game in that level) on a GFFX 5900Ultra. This demo has an ocean in its scene (a BIG ocean) and the entire ocean surface has two ps_1_1 shaders to effect, well, an ocean's surface.

I know exactly what those two ps_1_1 shaders codes are (unfortunately I can't provide them -- Ubisoft doesn't want me to) and those codes mean that "waves/ripples" should be rendered over the entire ocean's surface. Brent noticed that there are a number of spots where there are no "waves/ripples" on his 5900Ultra using 44.03s. Dave and I have confirmed that this "anomaly" happens only on 5600 and up but not on a 5200. I can confirm that both the demo and in actual gameplay in that level do not show up any such bug on a 5200 and Dave will investigate whether the bug exists in both the demo and in the gameplay level on a 5600 and above.

Sorry if you're asking for screenshots to depict what my attempt above to describe it -- I don't think it is right to provide these and fuel speculations until we get to the bottom of what's happening.

Dany Lepage (Ubisoft) has provided a very rough comment and I don't want to quote him in its entirety except that he thinks it is "definitely a driver bug". The thing is that we're trying to figure out why this "bug" exists on NV30/31/35 and not on NV34 even with NVIDIA's Unified Driver Architecture.

As for why Dave is highly suspicious, I have found out (and told Dave) that NVIDIA has a personnel focussed solely on SplinterCell benchmarking.

More, if there is more to be revealed, when we find more concrete information. For now, just take all of the above as no more than our providing our findings as we see them. I'd like to think it's a driver bug obviously but we read the above paragraph it's natural to become suspicious given the recent issues.

Few things here okay now I'm gonna be an *nasty word incase they ain't allowed* and say that kyle has been biased on this issue generally I like what he does but this time no. If kyle was to record a custom demo(s) for hardocp and keep them private how is the community supposed to know his scores are valid. Lets propose you have a game with levels that only partially use PS now if the demo record was to run around the level and view the only the parts without the pixel shaders and save it is that going to be even? I think not there are obivusly ways to make private demos strongly biased. Now even with private demos you can always just drop the IQ when timedemo are detected as stated by a number of people so its still not the answer. I'm sure publishers of games don't want the develpoers to be running around breaking cheating code in drivers they are wanting fixes/new games so its hard for game developers to stop this.

Brent has seemed to be completely opened minded about the whole thing I find its very sad that kyle has not said one word since the news has come out on his forum.
 
so lets just get down to the $500 question shall we?


who in their right mind would buy a 5900 ultra over a radeon 9800 pro (knowing that from what has been said) seeings that their drivers are


1. extremely buggy and therefore cannot correctly render features in your favorite games not giving you the IQ that you would EXPECT AND PAID FOR in that $500 card.

or

2. hacked beyond belief so that they can compete with the competition not giving you the IQ that you would EXPECT AND PAID FOR in that $500 card.

Funny how all the diehard nVidiots state "ATi drivers are crap and full of bugs" all the while it appears that nVidia drivers are far more "bug" ridden than anyone wants to admitt...

Personally this is the best part of all....when I play my games I get the same frame rates on the maps that the demos are recorded on for the most part. You will never get the exaxt same frames as your CPU is busy doing other things such as AI.........where in a time demo that part does NOT have to be done.
 
Reverend said:
As for why Dave is highly suspicious, I have found out (and told Dave) that NVIDIA has a personnel focussed solely on SplinterCell benchmarking.

:(

Reverend said:
I'd like to think it's a driver bug obviously but we read the above paragraph it's natural to become suspicious given the recent issues.

So very true, unfortunately. And with high school kids running nVidia's PR department these days I'm sorry to say they probably wont help rebuild any trust anytime soon. :(
 
Back
Top