ATI Benchmarking Whitepaper

Discussion in 'Graphics and Semiconductor Industry' started by Dave Baumann, Oct 15, 2003.

  1. Dave Baumann

    Dave Baumann Gamerscore Wh...
    Moderator Legend

    Joined:
    Jan 29, 2002
    Messages:
    14,079
    Likes Received:
    648
    Location:
    O Canada!
  2. Paul

    Veteran

    Joined:
    Feb 22, 2003
    Messages:
    1,974
    Likes Received:
    1
    Location:
    United States
    It's fine.
     
  3. micron

    micron Diamond Viper 550
    Veteran

    Joined:
    Feb 23, 2003
    Messages:
    1,189
    Likes Received:
    12
    Location:
    U.S.
    OMG they are becoming the aggressor :shock:
     
  4. demalion

    Veteran

    Joined:
    Feb 7, 2002
    Messages:
    2,024
    Likes Received:
    1
    Location:
    CT
    Interesting contrast to what has been proposed elsewhere, and it makes a great deal of sense from the consumer's perspective. Not exactly unexpected given their track record.

    I think what was in it was very good content. But I found one particular thing's absence (AFAIK, as I've mentioned before) a problem, at least if it isn't addressed in some document concerning ATI's cards that they target at reviewers:

    ...

    I'm still concerned about a lack of information about, or forced AF alternative to, the control panel AF behavior. There are indeed all sorts of reasons that make sense for having things work that way, but providing clear information about what is going on would be required to completely avoid misrepresentation.

    Of course, nVidia is doing worse :!: things with AF reduction overall, but with other IHVs entering the race, that's another matter. Besides that situation, there are new products that could easily be misrepresented in comparison to ATI cards. Such misrepresentation, if it occurs (i.e., new IHVs don't join the "texture stage shortcut" bandwagon, such that forced control panel AF comparisons are accurate), would be the fault of 1) the reviewer's ignorance of the issues, 2) each IHV who didn't accurately inform reviewers of what control panel AF meant, if it means something other than "applying a clear and standard AF and trilinear implementation to all texture stages".

    A repeat of my sentiments on the matter:

    I'm thinking it would be good if ATI didn't facilitate misrepresentation, by either targetting just "2" and adding something that clearly illustrates that the control panel forced AF isn't the highest possible quality for AF when "Quality" is checked (like adding another option for it, among other possibilities), or tackling "2" by nipping "1" in the bud, by simply making sure reviewers know what is going on for control panel/quality (in contrast to Application Preference) in documents like this :?: being sent out and discussing how to benchmark.

    It is apparent nVidia has been dedicated to grabbing the steering wheel, and forcefully driving people off a path to accurate performance comparison for quite a while. ATI doesn't try to grab the steering wheel, but their shortcut not being clearly marked is still a problem...even if you don't miss any scenery most of the time for it (because sometimes you do).
     
  5. nelg

    Veteran

    Joined:
    Jan 26, 2003
    Messages:
    1,557
    Likes Received:
    42
    Location:
    Toronto
    Common sense. To bad it needs to be repeated to review sites (B3D excluded). Furthermore it looks a little to much like an ad. for the 9600XT. There is an error on page four. It compares the 9600XT to a 5900Ultra, should be the 5600Ultra. I would like it to detail some of the specific optimizations that have been witnessed so some reviewers cough* Anand, Kyle and Tom * cough could have it as a handy reference.
     
  6. micron

    micron Diamond Viper 550
    Veteran

    Joined:
    Feb 23, 2003
    Messages:
    1,189
    Likes Received:
    12
    Location:
    U.S.
    [edit] Dave, you asked us what we think, but whats goin' through your head?
     
  7. Dean

    Newcomer

    Joined:
    May 26, 2003
    Messages:
    34
    Likes Received:
    0
    Location:
    Charlottetown, Canada
    I did not like the subtle threat "Do this or else be left behind" conclusion. Maybe its just me but I do not think they should tell a reviewer how to review. A reviewer should only take recommendations from his readership and not those who could financially profit from his recommendations.
     
  8. AlphaWolf

    AlphaWolf Specious Misanthrope
    Legend

    Joined:
    May 28, 2003
    Messages:
    8,562
    Likes Received:
    430
    Location:
    Treading Water
    I certainly didn't read it as a threat. And in general I would think very few readers have an informed opinion about benchmarking that they share with their favorite website.

    For the most part the document just seems to provide guidelines, which seem fairly reasonable. I think they would have been better served to leave the propaganda (xt benchmarks) out.
     
  9. ByteMe

    Banned

    Joined:
    May 27, 2003
    Messages:
    136
    Likes Received:
    7
    Only noticed the 5900ultra should be 5600ultra typo.
     
  10. madshi

    Regular

    Joined:
    Jul 26, 2002
    Messages:
    359
    Likes Received:
    0
    Apart from the 5600 vs 5900 typo I found it a good read. I'm not sure about the benchmarks. E.g. IIRC the ATI cards looked better at that 1280x1024 resolution vs the NVidia cards than in any other resolution with the current official drivers (no idea why). But I agree to all what was written in text. I like the pdf.
     
  11. Martin Eddy

    Regular

    Joined:
    Oct 5, 2003
    Messages:
    491
    Likes Received:
    4
    Location:
    Australia,Brisbane
    Are you guy's so sure the 5900 ultra thing was a typo?

    I mean look at the HL2 Benchmarks from Gabe Newells presentation.
     
  12. dan2097

    Regular

    Joined:
    May 23, 2003
    Messages:
    323
    Likes Received:
    0
    It all looked good to me except two things:

    The fx 5600 ultra/f5900 ultra typo

    The use of the Halo beta benchmark and half life 2 whatever stage it is benchmarks, those are to propoganda like for my liking. Sticking to the available games like tron 2.0 and tomb raider AOD shows the point
     
  13. Graphics_Krazy

    Newcomer

    Joined:
    Oct 15, 2003
    Messages:
    39
    Likes Received:
    1
    How does "If not, don't get left behind" translate to a threat?

    More than anything I think this doc was meant to wake up those reviewers still testing old "heavily optimized for benchmark" games. B3D of course not one of those sites.

    Also I think that was a typo too since the 5600Ultra benchmarks were followed by a sentence meant to conclude this.
     
  14. Hyp-X

    Hyp-X Irregular
    Veteran

    Joined:
    Feb 6, 2002
    Messages:
    1,170
    Likes Received:
    5
    Yes.
    The 9600Pro was faster than the 5900U but nowhere near as much as 5x.
    Also I think it was faster only in the pure DX9 path.
     
  15. Fred da Roza

    Newcomer

    Joined:
    May 6, 2003
    Messages:
    178
    Likes Received:
    2
    I am glad to see that it's posted here in the forums not the front page. Although I agree with the paper IMO companies don't seem to be able to write papers without some PR. All the selective benchmarks were a bit over the top. I don't blame them for that but B3D should not become a mouth piece for ATI. Articles like that should stay here in the forums.

    By the way who was the paper provided to, just B3D?
     
  16. Dave Baumann

    Dave Baumann Gamerscore Wh...
    Moderator Legend

    Joined:
    Jan 29, 2002
    Messages:
    14,079
    Likes Received:
    648
    Location:
    O Canada!
    I requested ATI if I could put this up in the forums as I thought it did actually bring up a few interesting points. The reason its "marketty" and tailored towards 9600XT is because this wen out on the review CD with the board, so I should imagine most other reviews have had it - and ignored it (I generally don't pay a great deal of attention to review guides but this one caught my eye).

    I think some of the points behind the age of benchmarks are a quite valid, if the most people ae actually playing newer, more advanced games - however whats possibly needed here is an independant gaming survey to actually undersatnd what games people are still playing (old reapeat titles / multiplayer titles) and what people intend to buy in the upcoming months. Whats scarier is the issues behind what ATI are saying is the disparity between popular benchmarks, and actual commonly used games, it raises the question whether reviews are actually any worth - Game PC's reviews indicate that there may be some truth to what they say.

    I think ATI need to be careful with the DX9 situation though as they may ben in danger of disappearing up their own backside here. While its clear they have the leg up in "default" DX9 performance, that won't necessarily represent the actual gaming landscape with alternate paths and its going to be tricky how that is handled. I also think they are in danger of making too many product decisions based on this percieved DX9 leadership - I already think that 9600 XT should have had somewhat faster RAM than it has shipped with, and I think its these shader decisions that are leading to these choices IMO.
     
  17. brent_anderson

    Newcomer

    Joined:
    Jul 22, 2003
    Messages:
    16
    Likes Received:
    0
    nevermind! duh!

    Brent
     
  18. THe_KELRaTH

    Regular

    Joined:
    Dec 9, 2002
    Messages:
    471
    Likes Received:
    0
    Location:
    Surrey Heath UK
    I thought it was well put together and quite correct, though I did have a chuckle when Fraps entered into the paper remembering what NVidia said.
    I'm always suprised when reviewers compare AA performance they always forget to mention that the NV3x is doing potentially half the workload when using the same driver settings.
    In the recent AT comparison review discussion in B3D many thought AA wasn't working on the NV35 rather than realising it performs Vertical offset AA only. This was found in the early NV30 reviews but since then has had little mention yet is an important factor when comparing with like for like driver settings. If we are to compare GPU processing power then surely NV3x should be set to x8aa v x4aa for ATI.
    .......

    I didn't realise there was such a big performance difference in BF which must be the most popular online game by quite a big margin (bar CS).
     
  19. Reverend

    Banned

    Joined:
    Jan 31, 2002
    Messages:
    3,266
    Likes Received:
    24
    This relates a great dela to what Patric/Futuremark wants to post at his own forum page here. But I'll leave that till when he posts it.

    In any case, I think everything about this paper is just plain common sense and the logical thing to do but it is a little bit difficult to implement for various reasons.

    For one, it is something we all want but rarely happens in reality and that is to have all reviewers (including those belonging to a single site) buy LOTSA games.

    I thought what they stated about FRAPS was already what everyone knows/wants but it is perhaps forgotten that for this to have "a few percentages differences" games needs to benchmarked by one single person using FRAPS in all the games that person uses. Logistically, it is very difficult for Dave to explain to Neeyik and me how he benchmarks, say, HALO, using FRAPS in a certain gameplay sequence and expect us to repeat that and expect the public to think that Dave, Neeyik and I benchmarks HALO exactly the same way when all our reviews (three reviews by three different authors all reviewing the same basic chip but by different vendors) are read by them. I hope you guys get what I mean. Especially when we cannot record demos of games.

    In any case, this paper is a ho-hum thing to read.
     
  20. Hellbinder

    Banned

    Joined:
    Feb 8, 2002
    Messages:
    1,444
    Likes Received:
    12
    Its Quite Clear now that Nexus read this and the Reviewer/Editors decided to do exactly as Far the opposite of this as they possibly Could.

    They Break or intentionally go against every single point. they even break new Ground where they can by using the unreleased Nv36.

    It becomes so Clear which sites are controled by Blind Nvidia Worshipers and which site like Fair revews.
     
Loading...

Share This Page

  • About Us

    Beyond3D has been around for over a decade and prides itself on being the best place on the web for in-depth, technically-driven discussion and analysis of 3D graphics hardware. If you love pixels and transistors, you've come to the right place!

    Beyond3D is proudly published by GPU Tools Ltd.
Loading...