ATI Benchmarking Whitepaper
- Thread starter Dave Baumann
- Start date
A good read and generally agreeable, and no, I don't read that particular comment as a "threat" either. In fact, it mainly points to what I've personally found out the more I've read reviews and flipped around the sites. I've only been really paying attention since, I figure, Dec 2002. Before that I was mainly just looking at Anand's or Tom's for quick looks at the products and not really delving deeper. At this point, however, I've read a ton and think the exact points brought up, and sites doing only "lip-service" reviews will really NOT get me coming back, nor putting their links up on forums, nor referencing them to friends... So in that respect they are certainly going to be "left behind" as they are not only not "standing out" but are notably behind the curve in delivering real information. It WILL cost them (or at least won't "gain them"), and it is certainly something to keep up with or discard entirely, especially since we already get "lip-service" reviews from the print magazines which have much more circulation.
For people really looking for reviews, the effort has to be there or our eyes wander elsewhere.
For people really looking for reviews, the effort has to be there or our eyes wander elsewhere.
It's a good read despite some of the annoying PR which sticks out like a sore thumb. Nonetheless the reason old games are benched I suspect is because a vast majority of people play them.
I see more people playing Quake 2 than Unreal Tournament 2003.
This should say a lot.
Then again, benchmarks should reflect performance on upcoming games as gamers and enthusiasts tend to want to buy what's the fastest thing out there for the most demanding game/s.
I see more people playing Quake 2 than Unreal Tournament 2003.
This should say a lot.
Then again, benchmarks should reflect performance on upcoming games as gamers and enthusiasts tend to want to buy what's the fastest thing out there for the most demanding game/s.
DaveBaumann said:
Here's a trade secret - use the the top 50 downloaded demos list at FilePlanet
Here's the top 25 as of today:
1. Savage Demo
2. R.C. Cars Demo
3. Call of Duty Demo
4. XIII US Multiplayer Demo
5. Robin Hood: Defender of the Crown Demo
6. Medal of Honor: Allied Assault Breakthrough Multiplayer Demo
7. Star Wars Jedi Knight: Jedi Academy Single-Player Demo
8. Battlefield 1942: Secret Weapons of WWII Demo
9. Lock On: Modern Air Combat Demo
10. Beyond Good and Evil Demo [European]
11. Battlefield: 1942 Multiplayer Demo
12. Pirate Hunter Demo
13. Unreal Tournament 2003 Demo v2206
14. TRON 2.0 Single & Multiplayer Demo
15. Commandos 3: Destination Berlin Demo
16. Lord of the Rings: The Return of the King Demo
17. Worms 3D Demo
18. The War of the Ring Single-Player Demo
19. Asheron's Call 2 Free Trial
20. XIII Demo
21. Deer Hunter 2004 Demo
22. Devil Whiskey Demo v1.2.1
23. Navy SEALs: Weapons of Mass Destruction Demo
24. Beyond Good and Evil Demo [US]
25. The Lord of the Rings: War of the Ring Multiplayer Closed Beta
K.I.L.E.R said:
Yes. It says you hang out with strange people.
Well, I must say I´m very concerned about the 9600XT reviewers guide (the document Dave posted is included in it). I mean the file 9600XT_BENCHMARK_GUIDE.pdf.
The findings they speak about may be correct but I don´t like the way in princible. The guide is like a pre-generated review including benchmarks, IQ results, conclusions etc. You can just copy/paste most of it an voilá you have your full article. This is too much IMHO. Their job is to deliver the products and the job of the press is to test it and write about it. What I also don´t like is the fact that most of the findings they write about were findings discovered by reviewers. If they like findings then they should simply put links to the articles and give the credit to the reviewers and publications. Then editors can read it from an independend source. In the end, what ATI tries is heavy manipulation of the press. As I said, the content in the documents may be correct but in principle it´s pure manipulation. They can give hints on what they think is important for their products and talk about the strengths of their products in certain areas, but the way they do it right now is way too much.
Lars
The findings they speak about may be correct but I don´t like the way in princible. The guide is like a pre-generated review including benchmarks, IQ results, conclusions etc. You can just copy/paste most of it an voilá you have your full article. This is too much IMHO. Their job is to deliver the products and the job of the press is to test it and write about it. What I also don´t like is the fact that most of the findings they write about were findings discovered by reviewers. If they like findings then they should simply put links to the articles and give the credit to the reviewers and publications. Then editors can read it from an independend source. In the end, what ATI tries is heavy manipulation of the press. As I said, the content in the documents may be correct but in principle it´s pure manipulation. They can give hints on what they think is important for their products and talk about the strengths of their products in certain areas, but the way they do it right now is way too much.
Lars
Doomtrooper
Veteran
Borsti said:
So you are saying Nvidias 5800 ERA PR sent to reviewers (you know your site where you claimed the discontinued, laughing stock of the internet 5800 Ultra the 'new KING'). Where there was some sites threatened to use 'certain' settings with 'certain' drivers if they wanted to continue to recieve review products.
Don't make me laugh any more than I am.
Joe DeFuria
Legend
Borsti said:
You think that PDF was heavy manipulation of the press?
1) What in principal do you actually disagree with in that paper? (What testing methods, types of programs, etc.)
2) Did ATI make following any of those guidelines a condition of reviewing their product?
Sounds more like ATI is trying to inform the press of what makes a good and thorough review. Sadly, you can lead a horse to water....
Such blatant manipulation of the press by ATI should really be discussed at length by the Good Dr himself, IMHO.
Just like they were calling the 52.14 a WHQL candidate when they gave them to Anand for a review... Just like they advised reviewers to bench with the 51.75 before making a 180 and saying that those drivers were nowhere near ready when IQ cheats got exposed... Seeing a pattern here ? Strange how ATI is "marketing evil" while when Nvidia, with a documented history of releasing special "benchmarketing drivers" to the press to spoil competitor's product launches gets yet another free pabst. Err, pass, sorry. I suppose you didn't bother running the Dets in the DirectX filtering test ?
As I hinted earlier in this thread, Futuremark's Patric Ojala has posted his thoughts on a very similar matter on his own developer forum page.
Patric on game benchmarks versus benchmark applications
Patric informed me (and Dave) that his thoughts has nothing to do with ATI's paper and that he had in fact written this before Beyond3D made ATI's paper available online.
PS. I think I'll start a new thread about this.
Patric on game benchmarks versus benchmark applications
Patric informed me (and Dave) that his thoughts has nothing to do with ATI's paper and that he had in fact written this before Beyond3D made ATI's paper available online.
PS. I think I'll start a new thread about this.
Borsti said:
And you wonder why people have a hard time believing your site is unbiased
. If you call this pdf manipulation what do you call the endless parade of Det. 5x drivers bandied about with promises of impending release. With conditions of use. How about nV promise to Kyle at [H] that “application preference†selected in drivers would do full trilinear. They really lived up to that promise . In fact it seems (since there is a lack of official drivers from nV) that not only did they renege on that promise they made the cheat a new feature for all DX games. Of course who could blame them when sites like yours seldom, if ever, take them to task.Joe DeFuria
Legend
What really bothers me about Lars' attitude is that while of course the ATI PDF is part of their marketing material, it doesn't contain anything conceptually wrong about benchmarking that I can see. So ATI is getting criticized for being, um, truthful about benchmarking? Yes, the "truth" is likely to be advantageous to ATI.
But there's a problem with that how?
Contrast this to nVidia putting out PR telling people to NOT use "synthetic" tests like 3DMark 03. (Only to go back to 3DMark's beta program months later.) When to anyone with any reasonable approach to benchmarking knows that synthetic benchmarks have their significant place in any well rounded review.
I just don't get the double standard.
But there's a problem with that how?
Contrast this to nVidia putting out PR telling people to NOT use "synthetic" tests like 3DMark 03. (Only to go back to 3DMark's beta program months later.) When to anyone with any reasonable approach to benchmarking knows that synthetic benchmarks have their significant place in any well rounded review.
I just don't get the double standard.
GraphixViolence
Newcomer
So you're saying that ATI pointing out strengths of its products which many reviewers (including yourself) have overlooked in the past is "heavy manipulation of the press"? Does the fact that the document came from ATI make its contents any less true? There's nothing stopping you from verifying their performance results, or benchmarking different games to see if the same pattern holds. Unless they forced you to repeat what's in the document as a condition for granting you a review sample, then you are perfectly free to agree or disagree with the contents as you wish. Of course, if you're going to disagree, I'm sure your readers would like to know the reasons why.Borsti said:
Just a thing, but it's a new thing to me.
I got my first official review card from ATi yesterday and it came with a "Reviewer's Guide" and a "Press Kit" CD, but I looked at 'em as guidelines or suggestions as best. I never got the impression that they were trying to tell me how to bench their card, they just seem to be trying to point out it's strengths. (Which is sort of their job, which I ain't gonna hold against them for trying. )
But when it comes down to what I'm going to put this card thru or how I present my findings I can't find ANY restrictions, so I just don't see what the biggy-D is.
I got my first official review card from ATi yesterday and it came with a "Reviewer's Guide" and a "Press Kit" CD, but I looked at 'em as guidelines or suggestions as best. I never got the impression that they were trying to tell me how to bench their card, they just seem to be trying to point out it's strengths. (Which is sort of their job, which I ain't gonna hold against them for trying.
)But when it comes down to what I'm going to put this card thru or how I present my findings I can't find ANY restrictions, so I just don't see what the biggy-D is.
Borsti said:
How can you possibly compare a published "reviewer's guide," which in the end amounts to only suggestions for reviewers--which they are free to ignore if they choose--with practices like providing pre-announced, pre-shipping hardware to reviewers along with pre-shipping drivers which *must* be used with that hardware as a condition of the "review"...? Or, the practice of providing "mystery" benchmark results to certain Internet sites, which they then publish? What about nVidia printing up information which suggests that all synthetic benchmarking is "bad" while at the same time working to get DX9 game benchmarks actually removed from shipping games as we saw with TR:AoD, or else cheating them as we saw with 3dMk03? That seems far more egregious and manipulative to me than this--not even in the same league. The silence on your end concerning Eidos pulling that benchmark from its game is deafening.
In short, what is there in this document which tells you how you may or may not review an ATi product? If you are contending that the "reviewer's guide" is what you *must follow* to review an ATi product, then I'd agree with you.
But if it's what it seems to be--a suggestion, and nothing else--I can't imagine how you could reach your conclusion. Are you implying that it might somehow be ATi's fault that a reviewer might be so lazy and incompetent that he would simply "cut & paste" from this document without doing his own research? What excuse does a reviewer have for doing something like that, regardless of which company's products he's reviewing? Seems to me your beef is really with lazy/incompetent hardware reviewers instead of with ATi. It sounds like to me that what you're actually frightened of is that some "reviewers" might actually listen to some of the things suggested.
