ATI Benchmarking Whitepaper

Interesting contrast to what has been proposed elsewhere, and it makes a great deal of sense from the consumer's perspective. Not exactly unexpected given their track record.

I think what was in it was very good content. But I found one particular thing's absence (AFAIK, as I've mentioned before) a problem, at least if it isn't addressed in some document concerning ATI's cards that they target at reviewers:

...

I'm still concerned about a lack of information about, or forced AF alternative to, the control panel AF behavior. There are indeed all sorts of reasons that make sense for having things work that way, but providing clear information about what is going on would be required to completely avoid misrepresentation.

Of course, nVidia is doing worse :!: things with AF reduction overall, but with other IHVs entering the race, that's another matter. Besides that situation, there are new products that could easily be misrepresented in comparison to ATI cards. Such misrepresentation, if it occurs (i.e., new IHVs don't join the "texture stage shortcut" bandwagon, such that forced control panel AF comparisons are accurate), would be the fault of 1) the reviewer's ignorance of the issues, 2) each IHV who didn't accurately inform reviewers of what control panel AF meant, if it means something other than "applying a clear and standard AF and trilinear implementation to all texture stages".

A repeat of my sentiments on the matter:

I'm thinking it would be good if ATI didn't facilitate misrepresentation, by either targetting just "2" and adding something that clearly illustrates that the control panel forced AF isn't the highest possible quality for AF when "Quality" is checked (like adding another option for it, among other possibilities), or tackling "2" by nipping "1" in the bud, by simply making sure reviewers know what is going on for control panel/quality (in contrast to Application Preference) in documents like this :?: being sent out and discussing how to benchmark.

It is apparent nVidia has been dedicated to grabbing the steering wheel, and forcefully driving people off a path to accurate performance comparison for quite a while. ATI doesn't try to grab the steering wheel, but their shortcut not being clearly marked is still a problem...even if you don't miss any scenery most of the time for it (because sometimes you do).
 
Common sense. To bad it needs to be repeated to review sites (B3D excluded). Furthermore it looks a little to much like an ad. for the 9600XT. There is an error on page four. It compares the 9600XT to a 5900Ultra, should be the 5600Ultra. I would like it to detail some of the specific optimizations that have been witnessed so some reviewers cough* Anand, Kyle and Tom * cough could have it as a handy reference.
 
I did not like the subtle threat "Do this or else be left behind" conclusion. Maybe its just me but I do not think they should tell a reviewer how to review. A reviewer should only take recommendations from his readership and not those who could financially profit from his recommendations.
 
I did not like the subtle threat "Do this or else be left behind" conclusion. Maybe its just me but I do not think they should tell a reviewer how to review. A reviewer should only take recommendations from his readership and not those who could financially profit from his recommendations.
Click to expand...

I certainly didn't read it as a threat. And in general I would think very few readers have an informed opinion about benchmarking that they share with their favorite website.

For the most part the document just seems to provide guidelines, which seem fairly reasonable. I think they would have been better served to leave the propaganda (xt benchmarks) out.
 
Apart from the 5600 vs 5900 typo I found it a good read. I'm not sure about the benchmarks. E.g. IIRC the ATI cards looked better at that 1280x1024 resolution vs the NVidia cards than in any other resolution with the current official drivers (no idea why). But I agree to all what was written in text. I like the pdf.
 
It all looked good to me except two things:

The fx 5600 ultra/f5900 ultra typo

The use of the Halo beta benchmark and half life 2 whatever stage it is benchmarks, those are to propoganda like for my liking. Sticking to the available games like tron 2.0 and tomb raider AOD shows the point
 
How does "If not, don't get left behind" translate to a threat?

More than anything I think this doc was meant to wake up those reviewers still testing old "heavily optimized for benchmark" games. B3D of course not one of those sites.

Also I think that was a typo too since the 5600Ultra benchmarks were followed by a sentence meant to conclude this.
 
madmartyau said:
Are you guy's so sure the 5900 ultra thing was a typo?
Click to expand...

Yes.
The 9600Pro was faster than the 5900U but nowhere near as much as 5x.
Also I think it was faster only in the pure DX9 path.
 
DaveBaumann said:
ATI have produced an interesting document on modern benchmarking practices, take a look and tell us what you think:
Click to expand...

I am glad to see that it's posted here in the forums not the front page. Although I agree with the paper IMO companies don't seem to be able to write papers without some PR. All the selective benchmarks were a bit over the top. I don't blame them for that but B3D should not become a mouth piece for ATI. Articles like that should stay here in the forums.

By the way who was the paper provided to, just B3D?
 
I requested ATI if I could put this up in the forums as I thought it did actually bring up a few interesting points. The reason its "marketty" and tailored towards 9600XT is because this wen out on the review CD with the board, so I should imagine most other reviews have had it - and ignored it (I generally don't pay a great deal of attention to review guides but this one caught my eye).

I think some of the points behind the age of benchmarks are a quite valid, if the most people ae actually playing newer, more advanced games - however whats possibly needed here is an independant gaming survey to actually undersatnd what games people are still playing (old reapeat titles / multiplayer titles) and what people intend to buy in the upcoming months. Whats scarier is the issues behind what ATI are saying is the disparity between popular benchmarks, and actual commonly used games, it raises the question whether reviews are actually any worth - Game PC's reviews indicate that there may be some truth to what they say.

I think ATI need to be careful with the DX9 situation though as they may ben in danger of disappearing up their own backside here. While its clear they have the leg up in "default" DX9 performance, that won't necessarily represent the actual gaming landscape with alternate paths and its going to be tricky how that is handled. I also think they are in danger of making too many product decisions based on this percieved DX9 leadership - I already think that 9600 XT should have had somewhat faster RAM than it has shipped with, and I think its these shader decisions that are leading to these choices IMO.
 
I thought it was well put together and quite correct, though I did have a chuckle when Fraps entered into the paper remembering what NVidia said.
I'm always suprised when reviewers compare AA performance they always forget to mention that the NV3x is doing potentially half the workload when using the same driver settings.
In the recent AT comparison review discussion in B3D many thought AA wasn't working on the NV35 rather than realising it performs Vertical offset AA only. This was found in the early NV30 reviews but since then has had little mention yet is an important factor when comparing with like for like driver settings. If we are to compare GPU processing power then surely NV3x should be set to x8aa v x4aa for ATI.
.......

I didn't realise there was such a big performance difference in BF which must be the most popular online game by quite a big margin (bar CS).
 
This relates a great dela to what Patric/Futuremark wants to post at his own forum page here. But I'll leave that till when he posts it.

In any case, I think everything about this paper is just plain common sense and the logical thing to do but it is a little bit difficult to implement for various reasons.

For one, it is something we all want but rarely happens in reality and that is to have all reviewers (including those belonging to a single site) buy LOTSA games.

I thought what they stated about FRAPS was already what everyone knows/wants but it is perhaps forgotten that for this to have "a few percentages differences" games needs to benchmarked by one single person using FRAPS in all the games that person uses. Logistically, it is very difficult for Dave to explain to Neeyik and me how he benchmarks, say, HALO, using FRAPS in a certain gameplay sequence and expect us to repeat that and expect the public to think that Dave, Neeyik and I benchmarks HALO exactly the same way when all our reviews (three reviews by three different authors all reviewing the same basic chip but by different vendors) are read by them. I hope you guys get what I mean. Especially when we cannot record demos of games.

In any case, this paper is a ho-hum thing to read.
 
Its Quite Clear now that Nexus read this and the Reviewer/Editors decided to do exactly as Far the opposite of this as they possibly Could.

They Break or intentionally go against every single point. they even break new Ground where they can by using the unreleased Nv36.

It becomes so Clear which sites are controled by Blind Nvidia Worshipers and which site like Fair revews.
 
You must log in or register to reply here.

Similar threads

Z
Its not just me, games used to be better
5 6 7
Replies
128
Views
5K
Shifty Geezer
Shifty Geezer
S
Retro - "We Are Killing Sony" - Leaked Docs Shine A Light On Sega's Ill-Fated Battle With PlayStation
Replies
11
Views
2K
stu11
S
E
Microsoft designed the 360 GPU?
Replies
10
Views
3K
davis.anthony
D
homerdog
Black artifacts/checkerboarding
Replies
13
Views
1K
homerdog
homerdog
M
Interesting old GPU architecture
Replies
17
Views
3K
Ailuros
Ailuros
Back
Top