Another Site reviwing with 3DMARK and 53.03

[CATALYST MAKER]

The only thing that gets my goat is when major hardware sites use drivers the public never see, year after year. Especially the ones who proclaim they'll never fall for that trick again, and then freakin' fall for it again.

We have never given review sites drivers that the public will not see as a CATALYST.

 

[John Reynolds]

You guys are being too hard on [H]/Brent. Look at Tweaktown's Halo scores and they're neck-to-neck. Look at [H]'s use of their custom Halo demo and ATI is 50% faster. Yes, Kyle is bullheaded when it comes to synthetics but at least they make a strong effort to circumvent cheating in their game benchmarking.

 

[digitalwanderer]

The only thing that gets my goat is when major hardware sites use drivers the public never see, year after year. Especially the ones who proclaim they'll never fall for that trick again, and then freakin' fall for it again.

We have never given review sites drivers that the public will not see as a CATALYST.

I don't think anyone was accusing ATi of doing that CM, but it's great to hear you speak up for ATi. (I've missed ya. )

ATi runs a class act since you've taken over their driver department Terry, I don't think I've ever heard anyone say otherwise. (Well, as long as we don't count ByteMe...but there's just no pleasing him. )

 

[digitalwanderer]

You guys are being too hard on [H]/Brent. Look at Tweaktown's Halo scores and they're neck-to-neck. Look at [H]'s use of their custom Halo demo and ATI is 50% faster. Yes, Kyle is bullheaded when it comes to synthetics but at least they make a strong effort to circumvent cheating in their game benchmarking.

And let's not forget to give Brent some big points for always being willing to come on the board here and explain/defend his positions, that ain't no small thing to be doing and it's a WHOLE lot better than most out there.

Brent really is alright, as long as you don't hold his employer against him. (And yes, I DID just get a very gross mansechs picture in my brain when I wrote that. )

 

[{Sniping}Waste]

This is from a thread for DriverHeaven for Zardon.

well until FM get a "certified driver" that works with this card, it will stay as is. im not passing on reviewing this hardware because there isnt a certified driver available from them. and if you had read the recent article on rojakpot you might wonder just how the whole concept works. The driver is WHQL.

http://www.rojakpot.com/ (bottom left) "Futuremark's Approved NVIDIA Drivers For 3DMark03!"

So really I dont personally class futuremarks classifications as 100% accurate either. Certainly not going on the educated reading ive done over the net in the last months.

"Futuremark asked me to remove the Rev. 53.05 results because they were not from a certified driver and presumably open to hanky-panky by NVIDIA's driver team. This apparently makes them unsuitable for comparisons against results from other GPUs.

But why recommend the Rev. 52.16 driver as the only certified NVIDIA driver? As they have stated on their website, the Rev. 52.16 drivers have
"3DMark03 specific optimization for the Pixel Shader 2.0 test and that score is solely comparable between NVIDIA cards"? The Rev. 52.16 drivers
clearly cannot be used to provide a valid and comparable 3DMark03 result."

If nvidia are cheating on 3dmark03 then they are cheating on everything we used in the review as the article as a whole shows the 5900XT beating the 9600 XT across the board. We studied the whole review and compared results between tests and the findings were that 3dmark03 in comparision to all the other test results were in line, so we felt there werent any specific 2003 optimisations, if there were then 3dmark03 would be vastly higher comparitively to all the other result figures.

We are looking into this driver set and have already contacted futuremark about it, so if I hear anything or have anymore findings ill be posting them. its just so messed up that we have to spend so much time trying to work out what is legit and what are cheats rather than reviewing the actual hardware itself. I really am also beginning to feel microsoft need to step in here and improve the WHQL video card driver process, so that when they give a set their certification they are at least sure the company involved whether its ATI, matrox or nvidia arent cheating on IQ for performance. Personally I feel a little part of this is down to Microsoft, after all who wants to pay $400 for a video card and find out the company who make it are making low IQ drivers to compete with a competitor and microsoft certification isnt stopping it?

I mean what can we do if NO other driver works on the 5900XT except a driver they supply which is incidentally approved by Microsoft? cancel the review on a hot new product which everyone is interested in? or do our best with what we have and then investigate all avenues to verify the findings. we included 6 games as well as synthetic benchmarks from several companies. We can still see in a game like NFSU which is heavily DX9 shader dependent the ATI product wins on half the tests.

To be honest, im seriously contemplating removing all FM benchmarks from our future reviews, its certainly not worth our while to be spending so much time looking into these benchmarks and the possible cheats involved with them.

Why review it against the 9600 XT? because they are both companies flagship product for mid range - one of the largest selling sectors. and incidentally the 5900XT is only £10 more here.

__________________
http://www.driverheaven.net/showthread.php?s=&postid=295083#post295083

 

[jimbob0i0]

You guys are being too hard on [H]/Brent. Look at Tweaktown's Halo scores and they're neck-to-neck. Look at [H]'s use of their custom Halo demo and ATI is 50% faster. Yes, Kyle is bullheaded when it comes to synthetics but at least they make a strong effort to circumvent cheating in their game benchmarking.

And let's not forget to give Brent some big points for always being willing to come on the board here and explain/defend his positions, that ain't no small thing to be doing and it's a WHOLE lot better than most out there.

Brent really is alright, as long as you don't hold his employer against him. (And yes, I DID just get a very gross mansechs picture in my brain when I wrote that. )

Yep he definately gets Kudos for that.... and their reviews *are* a lot fairer these days... now we have current games that push cards so it's not a case of "current games show no difference" but all the current games use last years tech and not 'soon to be current' tech...


Dig remember when Evan Lieb defended AT at nvnews over the separation into parts of reviews and the lack of IQ based comparison? We never did get any proper answers from him and he never returned so far as I have seen since that debacle.

 

[digitalwanderer]

Dig remember when Evan Lieb defended AT at nvnews over the separation into parts of reviews and the lack of IQ based comparison? We never did get any proper answers from him and he never returned so far as I have seen since that debacle.

I haven't heard nor seen of him since he got his ass handed to him last time he came around trying to tell us what we should think and why we were wrong to question Anand. 8)

But it's the example I was thinking of too, Brent isn't anything like that tool Lieb. Brent is still a geek who is willing to share information and if he's wrong he admits it and corrects it, that's a GOOD THING(tm) and something I admire the hell out of him for.

People like Lieb are the current bane of the hardware scene, IMHO.

 

[StealthHawk]

Is it really all that surprising that reviewers are using the latest NVIDIA driver(approved or not) with 3dmark03? What sane reviewer is going to go through the trouble to run one benchmark with 52.16? How is he/she going to get the point across the reader as to why this needed to be done? Even if the reason is articulated, will the reader even care?

How many actual users are going to use 52.16 when they run 3dmark? Do people care more about "valid" scores or "realistic real world" scores? IMO, the average person doesn't care about theoretical aspects of hardware. They simply want to know which card runs Game X faster. For whatever reason, 3dmark scores have been used as an overall indicator of how games will play, nevermind how accurate this metric really is.

 

[MasterBaiter]

Well.... looks like DH is dropping the use of FM from their benches, so there ya go!

"3dmarks dropped in DH reviews
Posted on Wednesday, February 11, 2004
at 10:33 PM by Zardon


I made a choice as the owner of this site to remove 3dmark from our last 5900 XT review, there will no more reviews on this site incorporating the 3dmarks as we spend more time debating, bickering and commenting on Futuremarks stance on drivers, whether nvidia are cheating or not etc, ad infinitum...... im not quoting or linking to the many editorials on numerous sites strewn all over the net on this subject. its getting old.

Reviewing hardware is a laborous process and rather than the actual overall content of our reviews (and the other tests we run of which there are might I add, quite a few) the topic of conversation always spirals into a Futuremark/nvidia debate - is it only me who is bored to tears reading this? I was always very pro Futuremark, but having read the many articles on the net over the last months I seriously doubt the validity of 3dmark benching, it could be strongly argued nvidia have ruined it as a useful tool - perhaps not, nonetheless unless we all wait for months after a new product is released and futuremark have had time to dissect the related driver its not a worthwhile exercise attempting to ascertain "cheats" when we could be doing more productive editorials for the site. A wide variety of real game tests will feature from this point onwards. "

 

[jimbob0i0]

Meh...

First person to claim the NV3X is great because it does great in 'current' games (citing UT2004 demo) and ignores the aspects of games *not* on last years engines I will personally hunt down

P.S> Yes UT2004 demo on max detail at 1600x1200 16xAF 6xAA (set via ini) does run GREAT on my XP3200+ 9700pro nf2 rig

 

[Pete]

Short-sightedness++, I guess.

Or, for the less naively optimistic, reality++.

Let's just hope DH puts the time it saves on 3DM bickering into providing useful benchmarks, like explaining:

1) what settings and shaders were used for Far Cry,
2) if the two cards looked the same, and
3) how often each hit its atrociously low minimum framerates.

Every review site should try to move to chronological fps graphs, IMO. If not in the review itself, at least as a way of double-checking their "one number no analysis" tendencies. But, again, this means more work for the time-pressed reviewer.

I'll echo the kudos for Brent. I like the direction their reviews have taken.

 

[3dilettante]

I'd like to see more sites drop 3dMark2k3, if only to expedite a better replacement.

It's not like wildebeast herds try to slow down to let a lame one keep up.
Let's face it, 3dMark2k3 is lame. Sure, people could back-rev the drivers to make card comparisons on drivers that are currently replaced, a far better solution is to chuck. If the thing is stuck on old drivers you are going to throw out, don't waste effort climbing into the driver time machine to keep alive a nightmare past its shelf date.

It might not be FutureMark's fault, but the fact remains that their product is compromised, and it is a waste of time trying to lock step with them in order to fend of Nvidia when they still roll over and get paid by said graphics company anyway.

Heck, for all we know, paying attention to FutureMark this long probably allowed Nvidia to slip cheats in on far more worthwhile programs. I'd like to see a quick taking of stock to see what we may be losing on important things.

 

[Mariner]

I wonder how much of a bonus the NV PR team get every time a website stops using 3DMark2003 in cases like this?

 

[Hanners]

What sane reviewer is going to go through the trouble to run one benchmark with 52.16?

Me?

Although you did say 'sane', so I guess that precludes me anyhow.

Well.... looks like DH is dropping the use of FM from their benches, so there ya go!

Yeah, I saw that too last night.

Isn't it funny how sites are happy to use 3DMark03 blindly, yet as soon as they get 'found out' and a wrist slap from FutureMark, they suddenly hate the benchmark and cry about how useless it is? You'd have thought that reviewers would welcome input and help (particularly certain sources), not rail against it and go into a tantrum whenever they are confronted with information about a mistake they have made. I should be finishing my first video card review of any note over the weekend, and I want people to pull it to pieces so I can make the next one better.

 

[Quitch]

You guys are being too hard on [H]/Brent. Look at Tweaktown's Halo scores and they're neck-to-neck. Look at [H]'s use of their custom Halo demo and ATI is 50% faster. Yes, Kyle is bullheaded when it comes to synthetics but at least they make a strong effort to circumvent cheating in their game benchmarking.

For once, I don't think this is directed at [H], it's directed at the attitude that 3DMark03 is BAD, which is backed up with a whole lot of rubbish. 3DMark03 is about the ONLY reliable benchmark out there, being the only benchmark making an effort to stop cheating. Yet, using drivers which can't cheat is too much effort, so reviewers drop it, and proceed to use games which are cheated on (perhaps only in the benchmarks at that) and think all is well because that annoying voice in the back of their head has gone.

The stupidity of it is mind boggling, and I think that's what annoys people here.

 

[andypski]

It's not worth our while as a review site attempting to make sure that people aren't cheating in benchmarks that we use. After all, it's hardly as if presenting correct and reliable benchmark results so that customers can make an informed choice about products is the sole purpose for our existence now is it?

Futuremark are trying to do all of the hard work for us by putting in the effort to approve drivers, but we don't trust them because it appears that they may have missed a few things in the past in the PS2.0 test, and hence they're apparently not completely infallible. Given this we simply can't rely on them to tell us which drivers are good or not - they might occasionally miss things!

Meanwhile our own extensive anti-cheating team consisting of a hamster and two gerbils has so far failed to find any cheats at all in any of our other benchmarks, so everything else is fine and this is just Futuremark's problem. (And believe me, we've had those rodents working night and day looking for those cheats, but all they managed to produce in that time was lots of poop.)

As a result of this thorough investigation we have decided to remove 3DMark03 from our benchmark suite because the Futuremark guys obviously have no idea what they're doing. I mean, just look at the facts - how can the creators of the benchmark possibly believe that they understand what's going on with their application better than our own team of highly trained professionals, and furthermore have the sheer gall to think that they can tell us which drivers are behaving correctly! It just doesn't add up!

In fact, we've decided that it may not be worth our while actually benchmarking cards in the future, because there's always the possibility that companies might be cheating despite the best efforts of our crack rodent team, so we have decided that for our next review we'll use a random number generator to produce the scores as it will save us a lot of time and effort that we might otherwise have to use in actually trying to verify our results. This is also great because none of the hardware companies can complain that this scheme is unfair because they'll have an exactly 50-50 chance of winning any given benchmark, and consumers can also be pleased because there's always at least some chance that our new RNG technique (catchy, huh?) will throw up a correct result from time to time, whereas any benchmark where cheats are being used would always be giving the wrong result - a clear improvement!

Now, we can't say fairer than that - our consciences are clear.

Doh - it's the Chewbacca defence!

Many review sites are going the extra mile these days and doing things like their own custom demos to try to improve their reviews - and this is certainly to be applauded and it's good to see these sorts of improvements.

On the other hand I just don't get the logic behind some people's stances on 3DMark03 - this idea that attempting to enforce certain rules in benchmarking is somehow worse than no enforcement at all just seems so counter-productive.

 

[nelg]

Ritire the rodents and send them this

 

[Veridian3]

1) what settings and shaders were used for Far Cry

All details were set to max quality in the drivers and in the game options.

2) if the two cards looked the same, and

Yes they did, there were no noticable differences in image quality.

3) how often each hit its atrociously low minimum framerates.

In Farcry the 59XT was slightly more playable than the 96XT, I wouldnt say either was acceptable however as far as drops go, very similar at this time.

Isn't it funny how sites are happy to use 3DMark03 blindly, yet as soon as they get 'found out' and a wrist slap from FutureMark, they suddenly hate the benchmark and cry about how useless it is? You'd have thought that reviewers would welcome input and help (particularly certain sources), not rail against it and go into a tantrum whenever they are confronted with information about a mistake they have made.

I wouldnt say we were found out. I am 100% happy with the decision to use 53.03 in that review. It is the only WHQL driver which works with the 59XT. If you install 52.16 on that card none of the required 3dmark03 features are supported. i.e. you get 2d use and thats it. That added to my statement that it is an uncertified driver which may contain optimisations should be enough for the reader to make their own decisions on the card and its performance.

Not to mention the only reason to do so is to make nVidia cards look better than they really are.

Absolutely not to make Nvidia look better, but to show what the card is capable of. If you look at 3dmark only then your statement would be true however there were 8 other benchmarks in that review, 5 of which were real world and 1 which was realworld/synthetic mixed (UT2003) and you should be able to see the overall picture which is very hard to skew.

 

[digitalwanderer]

I wouldnt say we were found out. I am 100% happy with the decision to use 53.03 in that review. It is the only WHQL driver which works with the 59XT. If you install 52.16 on that card none of the required 3dmark03 features are supported. i.e. you get 2d use and thats it. That added to my statement that it is an uncertified driver which may contain optimisations should be enough for the reader to make their own decisions on the card and its performance.

It's also a clear violation of the users agreement as well as an unfair way to show the performance difference between the cards Stu, and you're 100% happy with that decision? :|

Uhm, no. Wrong answer.

 

[karlotta]

________________________________________________________________
"I wouldnt say we were found out. I am 100% happy with the decision to use 53.03 in that review. It is the only WHQL driver which works with the 59XT. If you install 52.16 on that card none of the required 3dmark03 features are supported. i.e. you get 2d use and thats it. That added to my statement that it is an uncertified driver which may contain optimisations should be enough for the reader to make their own decisions on the card and its performance."
________________________________________________________________
And might not be a true indication of perfomance compared to other VPUs due to the fact FutureMark deams it unqualifide due to said "optimisations".
I wonder why the 59XT would run right with the 52.16? DUH!
And now Zardon wants to stop using the Mark.... its time to stop doing reviews of cards.. or just let neon do reviews....