Doom3 benches revisited.

Discussion in 'Architecture and Products' started by jjayb, May 19, 2003.

  1. Mephisto

    Newcomer

    Joined:
    Feb 7, 2002
    Messages:
    200
    Likes Received:
    0
    To summarize this a bit from my point of view:

    1) The D00M3 benchmarking was done on a system provided and prepared by NVIDIA in the NVIDIA Headquarters. (!)

    2) NVIDIA most probably knew about the issues in Catalyst 3.4.

    3) NVIDIA most probably spent a considerable amount of time tuning (and eventualy implementing cheats?) in their detonator FX drivers for the D00M III 0.02 alpha 3d engine, while ATI most probably did not.

    4) Borsti of RivaStation/Thomas Papst Hardware Guide admits that there have been massive time constraints. This makes it hard to verify the results or even detect possible "cheats" that result in lower image quality.

    5) As one result, Borsti admits that he benchmarked settings (HQ) that he basicly had no clue about what they force/apply/whatever.

    6) As a second result, he admits publishing wrong benchmark numbers on tomshardware.com.

    7) As a third result, it seems obvious to me that he wasn't fully aware of what is going on with a 256 MB ATI card with Catalyst 3.2.

    8.) The benchmarks run on a system with a CPU and system RAM not indicative for the average and peak CPU / system RAM performance available at the time id releases D00M3.

    9) Right now, nobody has the possibity to verify the results. Wrong and missleading results remain published until id desires they will allow the next run of benchmarks.

    Sad sad ...
     
  2. Mephisto

    Newcomer

    Joined:
    Feb 7, 2002
    Messages:
    200
    Likes Received:
    0
    So up to now, you haven't been aware of the difference between "application" and "quality"? Quite surprising! It is your JOB to find out what the differences are, to show the reader the quality difference between the modes and explain when they're applied when they select it in the driver tab. Obviously you don't know what the effects of driver settings are when you benchmark 3d graphic cards? Just timedemoing some settings is not "benchmarking" to me, it's playing around. One basic rule of benchmarking is beeing fully aware of the settings and their impact on IQ and performance, doing cross checking, verifing the results.
     
  3. GraphixViolence

    Newcomer

    Joined:
    Jul 3, 2002
    Messages:
    194
    Likes Received:
    2
    Because it hadn't been fully tested yet. If you're worried about stability in your public drivers, then why would you start including experimental optimizations for a game that's still several months away from release? It seems to me THAT would be silly... unless of course you've convinced the developer/publisher to release a surprise benchmark to catch your competitor off-guard.
     
  4. Pete

    Pete Moderate Nuisance
    Moderator Veteran

    Joined:
    Feb 7, 2002
    Messages:
    4,943
    Likes Received:
    347
    Again, I'm as much against what it appears nV did in 3DM03 as anyone (I've argued against it at Ars, AnandT, and here), but to carry that on to Doom 3 is a little much, IMO, simply because I think id has earned and maintained an impeccable reputation.

    Thanks for taking the time to explain, Lars. I'm still against early game numbers, because I think they're not totally representative, but if I were to take that to its logical conclusion, then I'd be against most previews on the 'net. I think I'd rather have the D3 numbers than not, but perhaps my biggest complaint is the prominence they were given (first benchmark at Tom's and Anand, and a separate article at [H]) considering how early the game, cards, and drivers used were. I'm not convinced at all that current D3 numbers can be used to predict future performance in D3 or in other games, but I'm very, very skeptical of many nV numbers ATM.
     
  5. Nite_Hawk

    Veteran

    Joined:
    Feb 11, 2002
    Messages:
    1,202
    Likes Received:
    35
    Location:
    Minneapolis, MN
    Lars:

    Good points, it shouldn't really be your job to make sure ATI or Nvidia have optimized drivers. In ATI's case, it's hard to tell what the deal is. It could probably go both ways. If ATI was told that Doom3 would start being previewed and reviewed by this time then yes, it's their own fault that their drivers arn't up to snuff. If on the other hand though, they weren't told that any kind of previews would be happening until a significantly later date, I'd be pretty pissed if I were them. Why should they work on optimizing their drivers for doom3 right now, when there are other games closer to release or already out that need optimizations? If they weren't expecting to be benchmarked for several months, I can see why they wouldn't have had any optimizations ready.

    I guess I am rather suprised that ID allowed the test to take place if they knew that ATI hadn't implemented any optimzations yet. JC isn't a stupied guy, and I'm sure he has an idea of how the engine and drivers should interact for good efficiency. There are a couple of scenarios that I can see. The first is that ID simply made a mistake. They either didn't think the tests would be that big of a deal, or trusted that if there were any problems that someone else would deal with making sure they got handled fairly. The second is that they didn't really care if things got represented fairly. Nvidia wanted a PR stunt, and ID let them have it. Whatever happened was (atleast to a certain extent) Nvidia's thing. Finally, perhaps JC has a good idea that any optimizations that ATI could make would at this point be rather trivial and not affect the benchmark in a major way. Perhaps his view is that the benchmark is actually a fair comparison of the cards.

    I guess we'll find out when review sites get a copy of the game that they can spend some time with themselves. Between the inaccuracies in reporting, inability to verify the tests, and the apparent one sided involvement in the benchmark, the whole thing is a big mess.

    Nite_Hawk
     
  6. WaltC

    Veteran

    Joined:
    Jul 22, 2002
    Messages:
    2,710
    Likes Received:
    8
    Location:
    BelleVue Sanatorium, Billary, NY. Patient privile
    OK, so Carmack's statement that the initial form of the demo was assembled by nVidia is correct...?

    By "us" of course you don't mean the general public, I would assume. It almost sounds as if you're saying nVidia noticed the "missing or wrong" textures and brought it to Carmack's attention...? But if so, why wasn't Carmack specific about why it was he "didn't like it" as nVidia presented it, and specific as to what changes he made? If your presumption about merely replacing textures is correct, there'd have been little to optimize in the drivers that hadn't already been done, right? Why should Carmack make a comment about nVidia's "courage" in "allowing" him to make his changes? Seems an odd comment indeed if the only changes he made were to a few textures.

    If, as you seem to be saying, there was nothing to this "demo" except a casually prepared timedemo based on code everybody else has including ATi--why not go ahead make a public release? What's the big deal? Again, if the demo is "good enough" to be used as a comparison for current and upcoming 3D cards, why would it fail to be "good enough" for a public release (as Carmack states he eventually intends to do)?

    Heh-Heh...then many of the "scores" should come as no surprise...;) I also heard from ATi that they'd heard nothing about this demo prior to its release as a part of nVidia's nv35 PR campaign. This would seem completely vindicated by the facts at hand.


    As well, NDA often prohibits me from answering questions for which I find few acceptable answers...;) (Mighty convenient, eh?) *chuckle* Kind of like Richard Nixon's use of "National Security" to decline from answering uncomfortable questions....Everybody can do that but in the end it really doesn't mean anything....It's not an answer, in other words.

    Really, though, who would suspect that there wouldn't be "anything different" between the leaked alpha and the current build of the software? That would be a mighty odd supposition for anyone to have, I think. People would surmise a difference without having to see a demo (and of course "people" still haven't seen such a demo), and for the public's edification ID could simply have released screen shots illustrating the differences--an nv35/R350 comparison would have nothing to do with that.

    As well, the leaked Alpha was leaked to the public whereas nobody "public" has this demo, so therefore I can't really see how handling the demo like this could be construed as some sort of "solution" to a largely exaggerated Doom III alpha "problem."

    Possibly, the demo ID deems suitable enough for public consumption won't come as a side-bar to a hardware release by some 3D-chip company--and we might see something that would constitute an equitable opportunity for all of the 3D-chip hardware players. That would be refreshing, if not novel these days.
     
  7. RussSchultz

    RussSchultz Professional Malcontent
    Veteran

    Joined:
    Feb 6, 2002
    Messages:
    2,855
    Likes Received:
    55
    Location:
    HTTP 404
    Christ, you people are in full witch hunt mode.
     
  8. Nite_Hawk

    Veteran

    Joined:
    Feb 11, 2002
    Messages:
    1,202
    Likes Received:
    35
    Location:
    Minneapolis, MN
    Russ:

    Would it have been worth it if they find a den of witches?

    Nite_Hawk
     
  9. jb

    jb
    Veteran

    Joined:
    Feb 6, 2002
    Messages:
    1,636
    Likes Received:
    7
    Lars,

    thanks for posting here. Weather your right/wrong is not my call. Always good to have more people here and hopefully we can all learn.
     
  10. WaltC

    Veteran

    Joined:
    Jul 22, 2002
    Messages:
    2,710
    Likes Received:
    8
    Location:
    BelleVue Sanatorium, Billary, NY. Patient privile
    *chuckle* OK, Russ, prove to me that witches don't exist....;)
     
  11. saf1

    Newcomer

    Joined:
    Nov 19, 2002
    Messages:
    235
    Likes Received:
    1
    You are wrong.

    It may have not changed anything in the NV35 review, true. However, the results painted a bleak picture for ATI. The results then lead to other sites posting. Then forums like this and others start hashing it out. Before you know it, yahoo has a top story stating how the NV35 is beating the ATI is DOOM3 results!!!

    While many of us here do not put much weight into news they put forth, you must admit it is a jump point for many browsers. When a headline show up as it did, people are bound to see it.

    And, it is very misleading.

    An honorable thing to have done is to have just ran the NV35 / Doom 3 tests and leave ATI out of it...
     
  12. Borsti

    Newcomer

    Joined:
    Feb 14, 2003
    Messages:
    91
    Likes Received:
    0
    So what? You think I can´t set up a system?

    Don´t know if they knew. Could they?

    Shame on ATI

    Detecting cheats was not a problem. I had time enough during the benchmark runs to compare quality

    I was unsure what certain quality settings did. I clarified that before I posted the results.

    Wrong. All numbers are absolutely correct. I never said anything like that.

    Ok.. .tell me what was going on with cat 3.2. Enlight us!

    interresting. What might be indicative in your eyes?

    They are neither wrong or missleading. Maybe they are just not showing you what you like to see?

    Lars
     
  13. jjayb

    Regular

    Joined:
    Feb 13, 2002
    Messages:
    358
    Likes Received:
    1
    Your doom3 high quality 4xaa bench's still show medium quality 4xaa numbers.

    You've given us the nv35 1024x768 high quality AA numbers here, what were the 9800's?

    You really should update the review though. As was pointed out the first page of medium quality numbers still says High quality at the top of the page also.
     
  14. FUDie

    Regular

    Joined:
    Sep 25, 2002
    Messages:
    581
    Likes Received:
    34
    There are many people who think they can, who actually can't.
    Do you even know if ATI has access to DOOM 3?
    The Catalyst 3.2 driver only recognizes 128 MB of the 256 MB Radeon 9800 Pro boards. Shame on you for not knowing or checking this.

    -FUDie
     
  15. Borsti

    Newcomer

    Joined:
    Feb 14, 2003
    Messages:
    91
    Likes Received:
    0
    That´s correct. As I said it´s a typo in the table. I´m working on the update.

    I did not run Quality AA on the Radeon.
     
  16. Borsti

    Newcomer

    Joined:
    Feb 14, 2003
    Messages:
    91
    Likes Received:
    0
    I think I really can set up a system... if I find out how to open that damn DVD drive ;-)

    Yes I think that ATI has access to Doom 3. They had in the past (http://www.ati.com/companyinfo/press/2002/4495.html) and they always say that they are working close with them. Last QuakeCon ATI provided the card to show Doom III (http://www.bluesnews.com/plans/6/)

    And on the last one: Yes, I checked if the driver is able to detect the 256MB. I did not expect that the driver says 256Mb while it´s only using 128. I will check that.
     
  17. FUDie

    Regular

    Joined:
    Sep 25, 2002
    Messages:
    581
    Likes Received:
    34
    Just like nvidia was working closely with FutureMark on 3D Mark 2003, until last December. Would you say that nvidia is still working closely with FutureMark just because they did in the past? There's a serious flaw in your logic.
    That's why god (MS) created the DX Caps Viewer in the SDK.

    -FUDie
     
  18. Borsti

    Newcomer

    Joined:
    Feb 14, 2003
    Messages:
    91
    Likes Received:
    0
    Thanks! I´m always learning as well. That´s why I take my time to look into forums :)

    Lars
     
  19. Borsti

    Newcomer

    Joined:
    Feb 14, 2003
    Messages:
    91
    Likes Received:
    0
    You say that I should stop using 3DM03 because NV left the beta program??
     
  20. Joe DeFuria

    Legend

    Joined:
    Feb 6, 2002
    Messages:
    5,994
    Likes Received:
    70
    It shuould change the conclusions considerably. Based on the "Wrong" labels, it looks the the NV35 does better on "medium settings without AA and without Aniso", and the NV35 ALSO does better when cranking up the image quality to maxium, plus AA.

    In short, based on the old and incorrect lables, the NV35 performs better at minimal AND maximum quality.

    NOW, with the correct labels, it looks like NV35 performs better in medium quality...but crank up the in-game quality, and the R350 pulls even. Dunno what ultimately happens when you crank up in-game quality AND AA settings. That paints a pretty different picture.

    Clarification, did you find a big difference in performance, or in Quality (or both?) I'll assume performance, because that's what I said Anand did not find all that different.

    I'll have to refute that though. According to your tests, The GeForceFX had a significant impact in performance when going from medium to high quality. (83 to 55 FPs at 1024). However, the Radeon did not. It only went from 68 to 61 FPS.

    Again, this is my point. Based on your numbers, the FX takes a major performance penalty when going from medium to high quality, and the Radeon 9800 does not.

    I don't know what you're trying to say here. The app calls for 8X aniso....how do you know there is a performance difference between what is asked for, vs. what is delivered? What is your baseline for measurement?

    Again, I don't follow you.

    Little performance difference between what? Do yo mean this:
    1) Turning off "forced" Aniso in the driver control panel, and letting Doom3 set the aniso level.

    and

    2) Turning on "Forced 8X Aniso" with default quality in the driver control panel....and running doom3 in what? High Quality or Medium Quality mode?

    You seem to be saying now (correct me if I'm wrong) that whether Aniso is forced by the panel, or from within Doom3 internally, the results are very close.

    If that's the case, then why is there any reason to suspect there are problems with Aniso in nVidia's drivers?

    Understood.

    That's good to hear.

    Their statements are to the contrary though in this case. (You can't post screen-shots, etc., because well, it's not "done" enough.)

    To be clear, I don't doubt that Carmack beileves that that HIS ENGINE CODE is pretty near final with only minor tweaks left. What about ATI's drivers though? Does he have ATI driver builds that the public does not? Does he know how much if any headroom is left in whatever ATI drivers he does have?

    I think that's a poor and misguided attitude to take. I would agree IF the benchark was either public or available to ATI and known to be publically tested with some notice. The point is, you have no idea if ATI has driver builds that are more Doom3 optimized than the cat 3.2s.

    The publically available drivers? Maybe, maybe not.

    Kudos to them. If they aren't ready, then they shouldn't do it.

    I wouldn't either, if all vendors had amply notice that the benchmarks were going to be done, to make sure that any driver optimisations they may have in house make it into the testing drivers.

    Shame on you.

    If one company knows that the game / demo is going to be released by date X, THEN shame on that company if they don't get drivers out to support it. If one company is blindsided by an unknown public display, SHAME ON ANYONE involved. This includes ID, NVIDIA, and yes, you.

    They were NOT aware of this public doom3 benchmarking.

    And if that is true, and if ATI doesn't get any Doom3 optimizations in their drivers by the time the test version THEY KNOW OF is released, then yes, shame on them.

    Shame on you for assuming that they don't have drivers in house that are not released to the public.

    :idea:

    Because your readers expect benchmarks to be done on as fair and level playing field as possible, perhaps :?:

    You seem to basically miss the premise that publically released drivers are not the same as in-house / development drivers. There's no sense in publically releasing drivers that have optimizations for a game or benchmark that is not available, if the suite of optimizations has not been tested enough to ensure it doesn't cause other problems to already shipping titles and benchmarks.

    That all depends on the circustances surrounding the test!

    It could be unfair to BOTH. That's the point.

    SILLY?!

    I explained it above, and someone else also explained it. Why SHOULD a company release drivers to the public that have optimizations for a game that doesn't (or shouldn't) exist yet!? Possible upside? NONE. Possible downside...negative effects for other games.
     
Loading...

Share This Page

  • About Us

    Beyond3D has been around for over a decade and prides itself on being the best place on the web for in-depth, technically-driven discussion and analysis of 3D graphics hardware. If you love pixels and transistors, you've come to the right place!

    Beyond3D is proudly published by GPU Tools Ltd.
Loading...