5900Ultra review at [H]ard OCP

[Nite_Hawk]

Out of curiosity, what would you guys think about a website that had a couple of reviews on popular websites reviews? Something of a peer-review site that would point out errors or descrepencies with other reviews? I'm thinking about content for a new display/3D tech site I'm working on.

Thanks,
Nite_Hawk

 

[SirPauly]

Not surprising really as the more popular commercial sites have offered questionable double standards, double talk and leaving out important information for years, imho. This truly is nothing new and will continue as long as they receive the hits, imho.

In a perfect world: I guess I would like to see the sites look out for the consumers more -- that's all.

 

[Ollo]

Thats an interesting idea, Nite_Hawk. I for one would like to see such a site.

 

[Tim Murray]

Out of curiosity, what would you guys think about a website that had a couple of reviews on popular websites reviews? Something of a peer-review site that would point out errors or descrepencies with other reviews? I'm thinking about content for a new display/3D tech site I'm working on.

Thanks,
Nite_Hawk

I was thinking about that very idea, but you run into problems.

1. Possible bias of the reviewer of the review?

2. The sheer amount of time it would take for one person to read every review (or most reviews) of a particular card and then compare those reviews against the reviews of others

3. How long before some site says you're stealing their content and then decides to go after you

Dunno, really. It's a good idea, but I don't think it would beat out the buyer reading several reviews himself in order to make a decision. Nor would it replace the boards that provide feedback for the mistakes in reviews (Brent and UT2003, Anand's 5900 review, etc.).

 

[Geo]

Out of curiosity, what would you guys think about a website that had a couple of reviews on popular websites reviews? Something of a peer-review site that would point out errors or descrepencies with other reviews? I'm thinking about content for a new display/3D tech site I'm working on.

Thanks,
Nite_Hawk

Great idea, but how are you going to fund it? To really be independant you need to not be getting your hardware for free, so that you aren't at the IHV's mercy, and --even more importantly-- to be seen by the public as not at their mercy.

 

[Nite_Hawk]

The Baron:

It'd be absolutely necessary to not play favorites with any particular review site or reivew. Still, the first point could be a problem. I think the other two problems wouldn't be too bad though. First, you really wouldn't need to copy any of the content from the reviews, only point out that a specific review's numbers varied by a certain amount compared to other reviews, and maybe comment on any potential complications like processor clockspeed, quality settings, cheating detection, etc. Copyright issues may come into play if you lifted portion of their review, but simply doing an analysis on their numbers and publishing your results should be ok. This would be a decent way to remain fairly objective too. What benchmarks did the game cover? Did they check for or mention anything about Cheat # X? etc.

Geo:

If I'm not doing hardware reviews, but peer-reviews on other site's reviews, then I don't have to be at the mercy of any vendor. You could argue bandwidth costs, but bandwidth seems to be cheap enough these days that I'm not too worried about it.

Nite_Hawk

 

[Slides]

How about something akin to a reviews version of www.resellerratings.com?

That way any allegations of bias can be minimized if some user input is also maintained.

 

[Geo]

Geo:

If I'm not doing hardware reviews, but peer-reviews on other site's reviews, then I don't have to be at the mercy of any vendor. You could argue bandwidth costs, but bandwidth seems to be cheap enough these days that I'm not too worried about it.

Nite_Hawk

How are you going to know if a review is fair/unfair, complete/incomplete, reproducible/unreproducible, etc if you don't have the hardware yourself to compare with and do your own testing?

 

[Nite_Hawk]

geo:

It'd be impossible to try and reproduce everyone's exact test conditions anyway. You'd have to build a machine with the same hardware, use the same OS, have the same drivers, etc etc. That's going to cost too much money, and take too much work. Instead, you could collect information about lots of different reviews, and create charts that show which ones used what drivers, systems, etc. and compare their clock speeds. You could comment on the quality settings they used, wether or not they accounted or checked for a certain cheat with the benchmark(s) that they ran.

The goal of this site wouldn't be to necessarily "prove" that other reviews are incorrect, but to collect information about many reviews and see deviations, find out who is doing their homework when testing (do they check for cheats? Do they use new drivers? Are their scores consistent accross reviews?) There is a lot of information out there that could be made useful.

Nite_Hawk

 

[Hanners]

It'd be impossible to try and reproduce everyone's exact test conditions anyway. You'd have to build a machine with the same hardware, use the same OS, have the same drivers, etc etc. That's going to cost too much money, and take too much work. Instead, you could collect information about lots of different reviews, and create charts that show which ones used what drivers, systems, etc. and compare their clock speeds. You could comment on the quality settings they used, wether or not they accounted or checked for a certain cheat with the benchmark(s) that they ran.

The goal of this site wouldn't be to necessarily "prove" that other reviews are incorrect, but to collect information about many reviews and see deviations, find out who is doing their homework when testing (do they check for cheats? Do they use new drivers? Are their scores consistent accross reviews?) There is a lot of information out there that could be made useful.

Nite_Hawk

This is my thoughts exactly.... So watch this space...

 

[Hanners]

It'd be impossible to try and reproduce everyone's exact test conditions anyway. You'd have to build a machine with the same hardware, use the same OS, have the same drivers, etc etc. That's going to cost too much money, and take too much work. Instead, you could collect information about lots of different reviews, and create charts that show which ones used what drivers, systems, etc. and compare their clock speeds. You could comment on the quality settings they used, wether or not they accounted or checked for a certain cheat with the benchmark(s) that they ran.

The goal of this site wouldn't be to necessarily "prove" that other reviews are incorrect, but to collect information about many reviews and see deviations, find out who is doing their homework when testing (do they check for cheats? Do they use new drivers? Are their scores consistent accross reviews?) There is a lot of information out there that could be made useful.

Nite_Hawk

These are my thoughts exactly.... So watch this space...

 

[Quinn1981]

Sigh. It appears that my IP is now banned at [H] forums.

*Hears the sound of large objects being swept under the carpet at [H]*

You could have been targeted because you work with us. If not, it certainly couldn't have helped.

Time for beer...

 

[Yonex]

Well they seem hard at work over there

It appears I too have been banned for asking serious, straightforward, no flamebait questions. No response just ban and block and evade.

The offending post.


The subtle changes.
It is most fascinating that we were watching the developments across the web regarding all the games that were going on with the Nvidia drivers.

We were told here to:
"wait and see".
"If you can't see it they aren't cheats."
They are optimizations.
The test is invalid.
It doesn't matter because Nvidia won't change.

Now Nvidia has even come out in THEIR OWN INTERNAL SLIDES detailing exact things they were accused of everywhere but here.

Kyle/Brent

Did Nvidia Cheat?
When did you know?
Why didn't you say anything?
Why aren't their any statements or corrections in the comparisions reflecting this in the current reveiws?
What about the IQ reduction that is still going on?

The biggest point here was Kyle's statement that he knew about this and that it wouldn't do any good to say anything about it because Nvidia wasn't going to change. That completely sells us the readers short.Well, it appears others disagreed and the pressure is in fact changing Nvidias behavior (see slides).

Was [H] wrong in the perception Nvidia wouldn't change?
Would you do things differently in the future? If so what?
Will [H] finally make a statement that the benches are false and try to get to the real numbers?

Please respond. My last set of questions was never answered.

 

[Fred da Roza]

Re: Well they seem hard at work over there

The biggest point here was Kyle's statement that he knew about this and that it wouldn't do any good to say anything about it because Nvidia wasn't going to change.

Agreed!!! Now that nVidia have apparently changed their minds and want to talk, Kyle is first in line with an editorial. It smells like marketing propaganda. Those prepared slides and "discussions" with [H} just reek of it. As I said to him (until my post was quickly deleted), an "editorial" would be more appropriate coming from someone who has been critical of nVidia's antics, like Beyond3D or Extreme Tech.

 

[Yonex]

I would agree

with you but I have been banned and posts deleted.

What kind of a small minded person cannot handle clearly thought out dissentting opinion?

 

[Fred da Roza]

Re: I would agree

with you but I have been banned and posts deleted.

What kind of a small minded person cannot handle clearly thought out dissentting opinion?

So have I.

First he says it won't change anything so there is no point in discussing it. All of a sudden that changes after his "discussions" with nVidia. How he justifies his silence and refusal to critisize nVidia with his about face "editorial" and nVidia PR slides is beyond me.

He once said to me that he knows he is doing the right thing because he is self aware. I couldn't believe he said that and responded accordingly. He clearly has backed himself into a corner and refuses to acknowledge any bias. As someone already said, his reputation is getting hammered and he is getting frustrated about it.

 

[WaltC]

I just wanted to say thanks to Dave for explaining and summerizing the situation for me. I do understand it better now.

I still want to do my own testing first of course, but I will see about putting an amend in the review to alert users of this issue and that we are looking into it.

Well, as long as you edit your review and tell readers that the results may be inaccurate and that you are looking into it, that is all we can ask for. And of course, you actually do look into it and reveal your findings, amending the conclusion of the article if that is warranted.

Unlike a certain site *cough*Anand*cough* who doesn't update their obviously incorrect graphs

I'm not sure what there is to "check into." This isn't rocket science, you know. Either the detail textures are trilineared or they aren't. If they aren't, this will inevitably increase performance. They don't appear to be doing anything but bilinear--open and shut. I just checked the review again and I see no mention of this issue. While correcting a major oversight like this is certainly better than nothing, it's still not as good as getting it right the first time (especially when you had knowledge of the situation prior to publication of the review and simply chose to ignore it.) And of course pledging to make amendments in a forum and then not doing so in the review is the equivalent of doing nothing.

And most of all, saying that (parphrased) "I didn't know there would be a performance difference between bi and tri-lineared detail textures in a game" is tantamount to saying one is unqualified to write such a review in the first place. I can't see much wiggle room here.

 

[StealthHawk]

I'm not sure what there is to "check into." This isn't rocket science, you know. Either the detail textures are trilineared or they aren't. If they aren't, this will inevitably increase performance. They don't appear to be doing anything but bilinear--open and shut. I just checked the review again and I see no mention of this issue. While correcting a major oversight like this is certainly better than nothing, it's still not as good as getting it right the first time (especially when you had knowledge of the situation prior to publication of the review and simply chose to ignore it.) And of course pledging to make amendments in a forum and then not doing so in the review is the equivalent of doing nothing.

Presumably Brent will have to post screenshots showing the IQ, as well as graphs showing the performance difference.

In a perfect world, it would be great if journalists didn't make mistakes or misrepresent the facts. Too often they fall short of our expectations. The next best thing is for them to admit they're wrong and bring the correct info to light. I can respect that. What I cannot respect is someone who refuses to admit they were wrong when the proof is handed to them, and who will not correct their mistakes by informing people of the truth. Furthermore, I cannot stand people who say that they will change but then don't. So far Brent sounds sincere and repentent. We will see if he is accountable or not. If there is another review where the issue is not brought to light my confidence in [H] reviews will drop dramatically.

 

[Atomahawk]

I was banned their yesterday for apperently posting off topic. But the fact of the matter is I questioned the fact that [H] had a “very candid conversation with the top brass at Nvidia â€￾ on FM03 but came back with nothing more than 3 ambiguous slides that admit no wrongdoing, and furthermore never asked about Shadermark, Fraps, FM01, Omega, and Nvidia’s claim that that 44.03 drivers are fully WHQL certified for the whole FX line when in fact only the 5900 is.

Making the value of their candid conversation totally useless considering nothing was brought back with [H] that would shed any light on any of these subjects, which good reporting would have done.

Of course he deleted that post, and never answered the following

--------------------------------------------------------------------------------
Originally posted by creedAMD
What did they admit screwing up?
--------------------------------------------------------------------------------
quote:

Originally posted by FrgMstr
Did you not read the slides? Is it not obvious to you? Please go back and read the problem statement and solution slides again.


My response (I'm not creedAMD but still responded )

Since we don't have a date as to when these slides were actually made, by whom, under what context and what the presenter said while making his presentation they carry absolutely no weight in your response to the question “did they admit to screwing upâ€￾ or benchmark doctoring as I call it. If your asking us to take this at face value, yes it can be perceived as such. Your asking us to take this at face value ? Why should we? Nvidia has admitted to nothing in those slides and furthermore has yet to even openly discuss on this or any other forum or there own web sight for that matter, their current position as it relates to the MULTIPLE instances of flagrant manipulation of current benchmarks used by reviewers and consumer as a basis prior to their decision to purchase a product from said company or affiliate.

To indicate that they have admitted to any benchmark doctoring or screwing up as you indicate is nothing short of putting words in Nvidia’s mouth. So until you can offer proof (as you so often say yourself) in a clearly written format from Nvidia you may want to refrain from misleading your readership into thinking that your opinion is actually fact.

And by the way I certainly don’t appreciate having a post that took me a better part of an hour to compose deleted, especially when we were invited to voice our opinion and followed your own guidelines. If your intent is to discourage me from doing so just indicate it clearly.

 

[swanlee]

Kyle will hopefully go down in flames soon. Notice how he completely avoids saying what he thinks Nvidia admitted to? Gees he is so lame, a shill like him deserves to be completely discredited and to be a lughing stock in the industry.

He has painted himself in a corner he didn't comment on the Nvidia problems because he said it would not do any good and Nvidia would not change yet these slides show they will change which completely kills the supposed reason why he was so quiet about the cheats.

I really hope he is not allowed to recover from this, if Nvidia is going to get away with years of cheating and consumer fraud i do hope the people that helped them get nailed.