What do you guys think of this Anand quote?

Those are the type of people that you should leave alone, so they can rip themselves off.
After all, it's a free country.

 

KHm... which one?

 

http://www.anandtech.com/weblog/index.html

 

Lol...

1) He has a "reliable source"....someone with HL2 access...
2) They have the "in" with nVidia...given the fact they were allowed to bench the NV38

I'll give 1 guess as to which company the "reliable source" works for...

 

You forget about the acces to the newest drivers, How else they get this score

 

Looks like the [H] / nVidia divorce is now complete.

It just saddens me that nVidia found a new b*tch in such short a time...

 

Err... Wasn't the 9800 supposed to be CPU-limited at 1280 anyway ?

And I agree, before :
- everyone gets access to those miracle drivers
- people apart from Anand get access to NV38 cards
- people get a chance to look at the various butcheatoptimizations performed

people shouldn't get too excited...

 

Followed your response from the other thread, Joe--Thanks.

Yes, who on earth might the "reliable source" be, if not nVidia? Who but nVidia would attempt to benefit from benchmark scores in HL2 contrasting ATi's products against nVidia's, running two different code paths?

I might imagine that a third party not nVidia would want to see a bunch more scores contrasting a bunch of different code path performances other than a noAA/noAF glimpse between the 5900U and the 9800P--and at a single resolution of 1024x768, no less. In short, I can see nothing of benefit Shimpi is providing to his readers with the publication of such information except as it is intended to promote an IHV--obviously, nVidia, in this case.

Personally, if Valve's source code has indeed been stolen, or else the "source code" is found to be fraudulent with derogatory notes in it about ATi, I believe if I was Valve I'd see about hitting Shimpi with a subpoena to force him to divulge his source for this "benchmark." It's possible there is a direct link to the "source code" stuff right there. Obviously, he has no intention of voluntarily releasing that information--which really makes you wonder why someone with a "reliable source" for such info, with nothing to hide, would mind naming his source. That makes Shimpi as guilty as his source as well as complicit in anything illegal that might be taking place here.

You know--a little sparring back and forth between companies in a restrained and civilized fashion, is one thing--but this is getting really nasty...I hate to say it but this has gone beyond honest competition, and I'm afraid we might be witnessing industrial espionage in its purest form here.

 

I had fun reading the weblog and the doubts about benchmarking with tombraiders. I bet tons more people are playing Tomb raider (despite not being a successfull title) than the total players of AM 1 to AM 3

I think working on including an EA game in the benchmark suite will be the next logical step to be fair to everyone

 

This is a question I've been wanting to ask for quite a while: What is the deal with web sites and Beta Drivers anyway? During the last year it should have become obvious to everyone that beta drivers are leaked on purpose and may or may not have serious issues with regards to quality vs. performance. I don't think anyone in their right minds can really claim that the correlation between beta drivers and final drivers would not be subject to possible changes. Yet to me it seems that many websites that I personally would categorize as respectable still jump way too easily to the 'beta drivers' bandwagon?

With beta drivers it's impossible for the readership to verify in any manner whether there are any issues or not. It's also virtually impossible for any website to do the required QA for testing these drivers (unless you happen to have a few weeks at your disposal and have capable engineers working for your website).

At the same time there are little if any reasons that I can think of why one could not wait until the public release (or even preferably the WHQL'd public release).

As far as I can tell the only reason to use beta drivers is the 'exclusivity' that you can get with them.

Could someone please explain to me why even very respectable websites so easily use unavailable, non-public beta drivers?

Cheers,

AJ

 

Because that what certain IHVs recommends they test with...and they don't want to "upset" the IHV for fear of retribution. Retribution being "you might not be on 'our list' in the future to get early access to hardware...so your web site is gonna lose some hits...."

I'm with you. I wish web sites would just not even entertain the thought of publishing benchmarks on drivers that are not yet available for public download from the vendor.

How about THIS for an ironic quote....I'll add emphasis:

http://www.anandtech.com/video/showdoc.html?i=1544&p=4

Perhaps hindsight isn't 20/20 for everyone...

 

AJ on that note (and I agree with you) this will stun you.....

Evan Lieb has been defending Anand's "review" of the NV38 and a few parts stand out (note I can't be bothered finding the right link again so I will paraphrase slightly although they are within the nvnews threads):

"Why use public drivers on a non-shipping card?"
"The performance enhancements seen in beta drivers often are included in final drivers so we might as well test the betas"
"Getting an exclusive on the NV38 is not about hits"

Hmm... in the style of Dogbert - BAH!

 

i believe that was in the nvnews forums, the Tom or the Anand thread in the nvidia cards part.

 

Okay - Aquamark 3 raised this same conumdrum - you have a game that can lower shader models from Dx9 all the way to Dx6 if your "DX9" class card can't cut it.

We are now told HL2 with NV35 use a mixed path - that sounds like a DX8 / DX9 mix rather than a pure DX9 path with Partial Precision hint spread liberally.

If this is the case what constitutes a fair analysis of a benchmark? If one card runs with much better effects and the other is faster with less effects what does a reviewer say? ATi gives you speed and effects, NVidia is slower with the same effects or faster without them. Trouble is then that you are left out trying to work out hard large the delats are in terms of both speed and effects.

 

The bottom of this page and the poor fellow gets shredded by the next one.

 

It looks unfortunate to me that in both that thread and one in the Anandtech forum that there isn't much discussion about the poor methodology behind using the deadly combination of unreleased video card, drivers, and prescott cpu.

In fact, in the case of the Anandtech thread, that is the one concern that is seemingly not given a proper excuse, though there are plenty of justifications with the layout of the article, the lack of information, etc.

I guess the lack of sound testing methodology is less flashy than supposed IHV bias and alleged selling out, but that's the one thing that I found disagreeable.

Should Anandtech have a bias, the next few articles will clearly show it, and I will give them the luxury of having a bias: I just won't listen to them.
However, I really don't like bad tests and the dubious conclusions drawn from them.

Putting together a list for the number of unknowns, we can see just how useful the review was.

Radeon 9800XT not being in general availability (fine if the article was only about that and it was the only unknown...)

NV38 not released. No specifications. Non-released drivers. No review of functionality. Could be faster than actuality, slower, equivalent, invalid.

Prescott not released. Few specifications. Unknown or limited performance data. No comparison to known processor, could be faster or slower or equivalent or possibly invalid.

System specs sparse. Unknown or limited performance data. No comparison to known setups, no reproducibility outside of Anandtech. Could be faster, slower, equivalent, invalid.

Test settings were not exhaustive or varied. Unknown or limited performance data. Due to lack of system data, unreproducible.
Could be faster, slower, equivalent, or invalid.

Test software of uncertain value. Granted there are few dx9 games out at the moment, it goes to the heart of the matter to test them. Software anomalies are quite possible on unreleased hardware and drivers.
Could be faster, slower, equivalent, or invalid.

There could be any combination of the above faster/slower/equivalent/invalid in respect to actuality(or what can be passed as a reasonable approximation), and there are five areas in this list alone. There could be additional areas and additional outcomes, but the numbers present are serious enough.

4^5 is 1024 unique combinations of valid and invalid points for this article, and vague hinting doesn't make it any more certain.

Unless they redo the entire thing and strip out a whole bunch of unknowns from each test run, the uncertainty is still there, and no excuse is going to change that. No amount of work or honorable intent is going to magically make up for a bad test basis, and this article had that to a great extent.

I don't give a rat's ass what IHV they like to cuddle with, but they sure better stop screwing with their tests.
Even biased tests tend to be more certain than what I've seen, since at least most are willing to show the systems they used.

At present, there is no restriction of unknowns, reproducibility, rigor in testing, nor transparency in the collection and representation of data. And no, saying "these numbers are great, but there may be image quality concerns which we might address later" does not count as being transparent.

edit: grammar/clarity

 

I tend to agree. As far as I can tell the way things typically pan out is that a manager from a major IHV gives you a call and urges you to use the 'latest drivers' because they give the 'correct' performance profile.

And well, you don't want to mess that relationship.

[/rant mode on] But from a journalistic point of view this is simply unbearable. I mean afterall any decent journalist that carries a responsibility towards his readership should simply run the performance figures with publicly available material and then make a note along the lines of 'please observe that these figures may change in the future when new drivers became available'.

Otherwise it's possible if not even likely that you're simply misleading your readership. But if you really, really want to do that there's no reason why you could run the default numbers with publicly available stuff and then include grey out versions of non-publics with the message and that in the future this might be a possible performance scenario (Lars Weinard has been doing this lately and I kind of like it too [keep it up Lars, you're doing the right thing! ]).

BTW: The same in my opinion goes for the benchmark tools also. There's now way that one can do proper technical due diligence on the benchmark tools and I don't think that it's very wise to jump into conclusion based on something that's not available to generic public (Doom III comes to mind, but this goes for HL2 also).

The part that I personally find quite frustrating is the fact that none of this is really rocked science. I find it constanty amazing that even major websites do not really seem to care much about their journalistic integrity and let the IHV's influence their practices.[/rant mode off]

Cheers,

AJ

 

*grin* Yeah, while you're at it, why not just print out any numbers that 'seem right' without any verification what so ever? Basically that's what it all boils down to. If you use testing software or drivers without being able to back up your claims then you could just as well print out numbers based on your own guess work and tell your readers that 'you have a hunch' that the performance might in the end look a bit like this.

*dry grin* Maybe I'm being a bit harsh here, but for heavens sake, there's so many very good alternatives that I'm just a tad annoyed with the 'we don't give a damn and why worry about integrity' approach to news that so many sites have.

The testing tools and the drivers being tested _have_ to be able to deal with public scutinity. Otherwise it's not really journalism, just fanboys flattering their idols...

Having said that I do think that there are constructive ways of dealing with issues. Lars Weindard's current "grey-out approach" is a very decent way of dealing with the issue and actually delivering more information to readers without misleading anyone. And yes, I do agree that HL2 performance as well as DoomIII performance are very important for the readership to know about and preferably before they've upgraded their machines. But this can also be executed properly with correct disclaimers and explanations so that the readers are not misled to any direction but rather can use that information as they see fit.

Cheers,

AJ