Doom3 benches revisited.

To summarize this a bit from my point of view:

1) The D00M3 benchmarking was done on a system provided and prepared by NVIDIA in the NVIDIA Headquarters. (!)

2) NVIDIA most probably knew about the issues in Catalyst 3.4.

3) NVIDIA most probably spent a considerable amount of time tuning (and eventualy implementing cheats?) in their detonator FX drivers for the D00M III 0.02 alpha 3d engine, while ATI most probably did not.

4) Borsti of RivaStation/Thomas Papst Hardware Guide admits that there have been massive time constraints. This makes it hard to verify the results or even detect possible "cheats" that result in lower image quality.

5) As one result, Borsti admits that he benchmarked settings (HQ) that he basicly had no clue about what they force/apply/whatever.

6) As a second result, he admits publishing wrong benchmark numbers on tomshardware.com.

7) As a third result, it seems obvious to me that he wasn't fully aware of what is going on with a 256 MB ATI card with Catalyst 3.2.

8.) The benchmarks run on a system with a CPU and system RAM not indicative for the average and peak CPU / system RAM performance available at the time id releases D00M3.

9) Right now, nobody has the possibity to verify the results. Wrong and missleading results remain published until id desires they will allow the next run of benchmarks.

Sad sad ...
 
Borsti said:
The reason I did not ran more HQ tests for the article was that I did not know what happened. There was a performance difference between the AF the app wants and the AF setting the driver is forcing. I spoke to JC about this and he thinks about solutions for that.

So up to now, you haven't been aware of the difference between "application" and "quality"? Quite surprising! It is your JOB to find out what the differences are, to show the reader the quality difference between the modes and explain when they're applied when they select it in the driver tab. Obviously you don't know what the effects of driver settings are when you benchmark 3d graphic cards? Just timedemoing some settings is not "benchmarking" to me, it's playing around. One basic rule of benchmarking is beeing fully aware of the settings and their impact on IQ and performance, doing cross checking, verifing the results.
 
Borsti said:
One can argue that there might be an optimization but it´s not implemented in the public drivers yet. But why should they make such a silly descission?
Because it hadn't been fully tested yet. If you're worried about stability in your public drivers, then why would you start including experimental optimizations for a game that's still several months away from release? It seems to me THAT would be silly... unless of course you've convinced the developer/publisher to release a surprise benchmark to catch your competitor off-guard.
 
Again, I'm as much against what it appears nV did in 3DM03 as anyone (I've argued against it at Ars, AnandT, and here), but to carry that on to Doom 3 is a little much, IMO, simply because I think id has earned and maintained an impeccable reputation.

Thanks for taking the time to explain, Lars. I'm still against early game numbers, because I think they're not totally representative, but if I were to take that to its logical conclusion, then I'd be against most previews on the 'net. I think I'd rather have the D3 numbers than not, but perhaps my biggest complaint is the prominence they were given (first benchmark at Tom's and Anand, and a separate article at [H]) considering how early the game, cards, and drivers used were. I'm not convinced at all that current D3 numbers can be used to predict future performance in D3 or in other games, but I'm very, very skeptical of many nV numbers ATM.
 
Lars:

Good points, it shouldn't really be your job to make sure ATI or Nvidia have optimized drivers. In ATI's case, it's hard to tell what the deal is. It could probably go both ways. If ATI was told that Doom3 would start being previewed and reviewed by this time then yes, it's their own fault that their drivers arn't up to snuff. If on the other hand though, they weren't told that any kind of previews would be happening until a significantly later date, I'd be pretty pissed if I were them. Why should they work on optimizing their drivers for doom3 right now, when there are other games closer to release or already out that need optimizations? If they weren't expecting to be benchmarked for several months, I can see why they wouldn't have had any optimizations ready.

I guess I am rather suprised that ID allowed the test to take place if they knew that ATI hadn't implemented any optimzations yet. JC isn't a stupied guy, and I'm sure he has an idea of how the engine and drivers should interact for good efficiency. There are a couple of scenarios that I can see. The first is that ID simply made a mistake. They either didn't think the tests would be that big of a deal, or trusted that if there were any problems that someone else would deal with making sure they got handled fairly. The second is that they didn't really care if things got represented fairly. Nvidia wanted a PR stunt, and ID let them have it. Whatever happened was (atleast to a certain extent) Nvidia's thing. Finally, perhaps JC has a good idea that any optimizations that ATI could make would at this point be rather trivial and not affect the benchmark in a major way. Perhaps his view is that the benchmark is actually a fair comparison of the cards.

I guess we'll find out when review sites get a copy of the game that they can spend some time with themselves. Between the inaccuracies in reporting, inability to verify the tests, and the apparent one sided involvement in the benchmark, the whole thing is a big mess.

Nite_Hawk
 
Borsti said:
The demo I tested with was recorded by Tim.

OK, so Carmack's statement that the initial form of the demo was assembled by nVidia is correct...?

There were still some missing or wrong textures in the demo level. id did not want us to see that. That´s why the demos were changed. My demo was recorded on saturday... not much time until sunday to optimize a driver for. But you´re right that we can´t be 100% sure that it happened that way.

By "us" of course you don't mean the general public, I would assume. It almost sounds as if you're saying nVidia noticed the "missing or wrong" textures and brought it to Carmack's attention...? But if so, why wasn't Carmack specific about why it was he "didn't like it" as nVidia presented it, and specific as to what changes he made? If your presumption about merely replacing textures is correct, there'd have been little to optimize in the drivers that hadn't already been done, right? Why should Carmack make a comment about nVidia's "courage" in "allowing" him to make his changes? Seems an odd comment indeed if the only changes he made were to a few textures.

If, as you seem to be saying, there was nothing to this "demo" except a casually prepared timedemo based on code everybody else has including ATi--why not go ahead make a public release? What's the big deal? Again, if the demo is "good enough" to be used as a comparison for current and upcoming 3D cards, why would it fail to be "good enough" for a public release (as Carmack states he eventually intends to do)?

NVIDIA and ATI always tell us that they are working very close with id. I was very surprised to hear from ATI that there´s no optimiaztion for Doom III implemented in their drivers yet.

Heh-Heh...then many of the "scores" should come as no surprise...;) I also heard from ATi that they'd heard nothing about this demo prior to its release as a part of nVidia's nv35 PR campaign. This would seem completely vindicated by the facts at hand.


I would like to talk on that as well but I would violate NDA information so I cant :(

As well, NDA often prohibits me from answering questions for which I find few acceptable answers...;) (Mighty convenient, eh?) *chuckle* Kind of like Richard Nixon's use of "National Security" to decline from answering uncomfortable questions....Everybody can do that but in the end it really doesn't mean anything....It's not an answer, in other words.

One reason for id was that many people have performance numbers from the leaked Alpha in mind. They wanted to show that reality is different. id was also VERY carefully this time. They did not want to see a new leak again. The game was dongled as well.

Really, though, who would suspect that there wouldn't be "anything different" between the leaked alpha and the current build of the software? That would be a mighty odd supposition for anyone to have, I think. People would surmise a difference without having to see a demo (and of course "people" still haven't seen such a demo), and for the public's edification ID could simply have released screen shots illustrating the differences--an nv35/R350 comparison would have nothing to do with that.

As well, the leaked Alpha was leaked to the public whereas nobody "public" has this demo, so therefore I can't really see how handling the demo like this could be construed as some sort of "solution" to a largely exaggerated Doom III alpha "problem."

We´ll see how usefull those results were when the first demo will be released.

Possibly, the demo ID deems suitable enough for public consumption won't come as a side-bar to a hardware release by some 3D-chip company--and we might see something that would constitute an equitable opportunity for all of the 3D-chip hardware players. That would be refreshing, if not novel these days.
 
Lars,

thanks for posting here. Weather your right/wrong is not my call. Always good to have more people here and hopefully we can all learn.
 
Borsti said:
I now know that I could have done things better and I learned from it. id wanted to show some Doom III numbers. From what I learned from id´s politics so far is that they do not publish anything until it´s done. So I still trust in those numbers. JC would´nt have allowed this if there´s still trouble with the codepaths. But in the end it did not change anything in the NV35 review. Those DIII numbers were just a bonus. There´s no word in my final conclusion about it.
Lars

You are wrong.

It may have not changed anything in the NV35 review, true. However, the results painted a bleak picture for ATI. The results then lead to other sites posting. Then forums like this and others start hashing it out. Before you know it, yahoo has a top story stating how the NV35 is beating the ATI is DOOM3 results!!!

While many of us here do not put much weight into news they put forth, you must admit it is a jump point for many browsers. When a headline show up as it did, people are bound to see it.

And, it is very misleading.

An honorable thing to have done is to have just ran the NV35 / Doom 3 tests and leave ATI out of it...
 
Mephisto said:
To summarize this a bit from my point of view:

1) The D00M3 benchmarking was done on a system provided and prepared by NVIDIA in the NVIDIA Headquarters. (!)

So what? You think I can´t set up a system?

2) NVIDIA most probably knew about the issues in Catalyst 3.4.

Don´t know if they knew. Could they?

3) NVIDIA most probably spent a considerable amount of time tuning (and eventualy implementing cheats?) in their detonator FX drivers for the D00M III 0.02 alpha 3d engine, while ATI most probably did not.

Shame on ATI

4) Borsti of RivaStation/Thomas Papst Hardware Guide admits that there have been massive time constraints. This makes it hard to verify the results or even detect possible "cheats" that result in lower image quality.

Detecting cheats was not a problem. I had time enough during the benchmark runs to compare quality

5) As one result, Borsti admits that he benchmarked settings (HQ) that he basicly had no clue about what they force/apply/whatever.

I was unsure what certain quality settings did. I clarified that before I posted the results.

6) As a second result, he admits publishing wrong benchmark numbers on tomshardware.com.

Wrong. All numbers are absolutely correct. I never said anything like that.

7) As a third result, it seems obvious to me that he wasn't fully aware of what is going on with a 256 MB ATI card with Catalyst 3.2

Ok.. .tell me what was going on with cat 3.2. Enlight us!

8.) The benchmarks run on a system with a CPU and system RAM not indicative for the average and peak CPU / system RAM performance available at the time id releases D00M3.

interresting. What might be indicative in your eyes?

9) Right now, nobody has the possibity to verify the results. Wrong and missleading results remain published until id desires they will allow the next run of benchmarks.

They are neither wrong or missleading. Maybe they are just not showing you what you like to see?

Lars
 
Wrong. All numbers are absolutely correct. I never said anything like that.

Your doom3 high quality 4xaa bench's still show medium quality 4xaa numbers.

You've given us the nv35 1024x768 high quality AA numbers here, what were the 9800's?

You really should update the review though. As was pointed out the first page of medium quality numbers still says High quality at the top of the page also.
 
Borsti said:
Mephisto said:
To summarize this a bit from my point of view:

1) The D00M3 benchmarking was done on a system provided and prepared by NVIDIA in the NVIDIA Headquarters. (!)
So what? You think I can´t set up a system?
There are many people who think they can, who actually can't.
3) NVIDIA most probably spent a considerable amount of time tuning (and eventualy implementing cheats?) in their detonator FX drivers for the D00M III 0.02 alpha 3d engine, while ATI most probably did not.
Shame on ATI
Do you even know if ATI has access to DOOM 3?
7) As a third result, it seems obvious to me that he wasn't fully aware of what is going on with a 256 MB ATI card with Catalyst 3.2
Ok.. .tell me what was going on with cat 3.2. Enlight us!
The Catalyst 3.2 driver only recognizes 128 MB of the 256 MB Radeon 9800 Pro boards. Shame on you for not knowing or checking this.

-FUDie
 
jjayb said:
Wrong. All numbers are absolutely correct. I never said anything like that.

Your doom3 high quality 4xaa bench's still show medium quality 4xaa numbers.

You've given us the nv35 1024x768 high quality AA numbers here, what were the 9800's?

You really should update the review though. As was pointed out the first page of medium quality numbers still says High quality at the top of the page also.

That´s correct. As I said it´s a typo in the table. I´m working on the update.

I did not run Quality AA on the Radeon.
 
FUDie said:
Borsti said:
Mephisto said:
To summarize this a bit from my point of view:

1) The D00M3 benchmarking was done on a system provided and prepared by NVIDIA in the NVIDIA Headquarters. (!)
So what? You think I can´t set up a system?
There are many people who think they can, who actually can't.
3) NVIDIA most probably spent a considerable amount of time tuning (and eventualy implementing cheats?) in their detonator FX drivers for the D00M III 0.02 alpha 3d engine, while ATI most probably did not.
Shame on ATI
Do you even know if ATI has access to DOOM 3?
7) As a third result, it seems obvious to me that he wasn't fully aware of what is going on with a 256 MB ATI card with Catalyst 3.2
Ok.. .tell me what was going on with cat 3.2. Enlight us!
The Catalyst 3.2 driver only recognizes 128 MB of the 256 MB Radeon 9800 Pro boards. Shame on you for not knowing or checking this.

-FUDie

I think I really can set up a system... if I find out how to open that damn DVD drive ;-)

Yes I think that ATI has access to Doom 3. They had in the past (http://www.ati.com/companyinfo/press/2002/4495.html) and they always say that they are working close with them. Last QuakeCon ATI provided the card to show Doom III (http://www.bluesnews.com/plans/6/)

And on the last one: Yes, I checked if the driver is able to detect the 256MB. I did not expect that the driver says 256Mb while it´s only using 128. I will check that.
 
Borsti said:
Yes I think that ATI has access to Doom 3. They had in the past (http://www.ati.com/companyinfo/press/2002/4495.html) and they always say that they are working close with them. Last QuakeCon ATI provided the card to show Doom III (http://www.bluesnews.com/plans/6/)
Just like nvidia was working closely with FutureMark on 3D Mark 2003, until last December. Would you say that nvidia is still working closely with FutureMark just because they did in the past? There's a serious flaw in your logic.
And on the last one: Yes, I checked if the driver is able to detect the 256MB. I did not expect that the driver says 256Mb while it´s only using 128. I will check that.
That's why god (MS) created the DX Caps Viewer in the SDK.

-FUDie
 
jb said:
Lars,

thanks for posting here. Weather your right/wrong is not my call. Always good to have more people here and hopefully we can all learn.

Thanks! I´m always learning as well. That´s why I take my time to look into forums :)

Lars
 
Just like nvidia was working closely with FutureMark on 3D Mark 2003, until last December. Would you say that nvidia is still working closely with FutureMark just because they did in the past? There's a serious flaw in your logic.

You say that I should stop using 3DM03 because NV left the beta program??
 
Borsti said:
My results are correct - as far as I can say that. I messed up the table description but that does not change anything on the results (numbers).

It shuould change the conclusions considerably. Based on the "Wrong" labels, it looks the the NV35 does better on "medium settings without AA and without Aniso", and the NV35 ALSO does better when cranking up the image quality to maxium, plus AA.

In short, based on the old and incorrect lables, the NV35 performs better at minimal AND maximum quality.

NOW, with the correct labels, it looks like NV35 performs better in medium quality...but crank up the in-game quality, and the R350 pulls even. Dunno what ultimately happens when you crank up in-game quality AND AA settings. That paints a pretty different picture.

I don´t know why Anand said that there´s no difference between HQ or medium. You have to ask him. I found a big difference between those settings.

Clarification, did you find a big difference in performance, or in Quality (or both?) I'll assume performance, because that's what I said Anand did not find all that different.

I'll have to refute that though. According to your tests, The GeForceFX had a significant impact in performance when going from medium to high quality. (83 to 55 FPs at 1024). However, the Radeon did not. It only went from 68 to 61 FPS.

Again, this is my point. Based on your numbers, the FX takes a major performance penalty when going from medium to high quality, and the Radeon 9800 does not.

There was a performance difference between the AF the app wants and the AF setting the driver is forcing. I spoke to JC about this and he thinks about solutions for that.

I don't know what you're trying to say here. The app calls for 8X aniso....how do you know there is a performance difference between what is asked for, vs. what is delivered? What is your baseline for measurement?

The default AF rendering in Detonator FX is Quality - as well as in ATIs driver. There was a little difference between HQ and the AF driver settings (about 0,5 to 0,2FPS depending on the resolution). So HQ shows correct numbers of the high quality AF modes of both cards.

Again, I don't follow you.

Little performance difference between what? Do yo mean this:
1) Turning off "forced" Aniso in the driver control panel, and letting Doom3 set the aniso level.

and

2) Turning on "Forced 8X Aniso" with default quality in the driver control panel....and running doom3 in what? High Quality or Medium Quality mode?

You seem to be saying now (correct me if I'm wrong) that whether Aniso is forced by the panel, or from within Doom3 internally, the results are very close.

If that's the case, then why is there any reason to suspect there are problems with Aniso in nVidia's drivers?

One other reason why I did not more HQ testings was simply time. I thought it´s more important to run tests with as many cards I can. And it takes a lot time to run these tests. I had to decide between detailed tests with two cards or running as many cards I can. In the end I did a mix of both.

Understood.

There were also no rules from NV that we had to test specific quality modes only or anything else.

That's good to hear.

From what I learned from id´s politics so far is that they do not publish anything until it´s done.

Their statements are to the contrary though in this case. (You can't post screen-shots, etc., because well, it's not "done" enough.)

So I still trust in those numbers. JC would´nt have allowed this if there´s still trouble with the codepaths.

To be clear, I don't doubt that Carmack beileves that that HIS ENGINE CODE is pretty near final with only minor tweaks left. What about ATI's drivers though? Does he have ATI driver builds that the public does not? Does he know how much if any headroom is left in whatever ATI drivers he does have?

I know that ATI feels handicaped here. But it´s not my fault that they did´nt optimize the driver yet (as they say).

I think that's a poor and misguided attitude to take. I would agree IF the benchark was either public or available to ATI and known to be publically tested with some notice. The point is, you have no idea if ATI has driver builds that are more Doom3 optimized than the cat 3.2s.

I also know that ATI would have liked to do the same thing with Halflife2, since it´s developed on ATI cards and the drivers are allready optimized for the game.

The publically available drivers? Maybe, maybe not.

The reason why that did not happen was that Valve did not like to do this (yet).

Kudos to them. If they aren't ready, then they shouldn't do it.

I would´nt have any problem in running HL2 benchmarks at this time.

I wouldn't either, if all vendors had amply notice that the benchmarks were going to be done, to make sure that any driver optimisations they may have in house make it into the testing drivers.

If the game developer says that they´re ready for a performance evaluation there´s no reason not to do that. If one company neglected their optimization for a certain game yet - well, shame on them!

Shame on you.

If one company knows that the game / demo is going to be released by date X, THEN shame on that company if they don't get drivers out to support it. If one company is blindsided by an unknown public display, SHAME ON ANYONE involved. This includes ID, NVIDIA, and yes, you.

ATI is well aware of the development status of Doom III.

They were NOT aware of this public doom3 benchmarking.

They also know that we´ll see test version of it in the near future.

And if that is true, and if ATI doesn't get any Doom3 optimizations in their drivers by the time the test version THEY KNOW OF is released, then yes, shame on them.

So shame on ATI if they did not optimize their driver yet.

Shame on you for assuming that they don't have drivers in house that are not released to the public.

I don´t want to take care if it fits into the marketing of one company if run benchmarks with a game, even if it´s not yet released. Why should I take care?

:idea:

Because your readers expect benchmarks to be done on as fair and level playing field as possible, perhaps :?:

Everybody says that they are working very close with the game developers but if you run some benchmark everything is suddenly different?

You seem to basically miss the premise that publically released drivers are not the same as in-house / development drivers. There's no sense in publically releasing drivers that have optimizations for a game or benchmark that is not available, if the suite of optimizations has not been tested enough to ensure it doesn't cause other problems to already shipping titles and benchmarks.

Let´s take HL2 as an (virtual) example. If you want to buy a new card right now and you´re waiting for that game. If I run benchmarks at this time with an alpha the ATI cards might look better. So what´s the conclusion for that guy? He´ll buy an ATI card. Is that unfair?

That all depends on the circustances surrounding the test!

Unfair for whom? For NV or for the consumer?

It could be unfair to BOTH. That's the point.

One can argue that there might be an optimization but it´s not implemented in the public drivers yet. But why should they make such a silly descission?

SILLY?!

I explained it above, and someone else also explained it. Why SHOULD a company release drivers to the public that have optimizations for a game that doesn't (or shouldn't) exist yet!? Possible upside? NONE. Possible downside...negative effects for other games.
 
Back
Top