Gamersdepot with a review with 3DMARK 03 and 53.03

[MrGaribaldi]

]

I'm not quite sure how much more extra work for the reviewers it would be to download a list of approved/unapproved drivers, considering they're allready downloading the latest drivers from the different IHV's...

..or just check the Approved Drivers page for information on which drivers are approved? If that's something some find irritating, why would some update which tells them the same thing only in the software itself be any different?

Because of the mindset that, imo, most reviewers and their audience has: Always update the software you are using to the latest version.

Also see AzBats answer.

]

And if they don't have the computer they test on connected to the net, they still have to copy the drivers for the card they're testing, at which point a manual download of the list of drivers shouldn't be too hard to do at the same time...

But when they start the benchmark (assuming it would have the auto-updater) it checks for updates. Then they already have all drivers installed, and most likely would have unplugged the net. As I said above, I am not sure what big difference there would be if they would A) need to download manually an update + install it just to get the info on what's approved and what's not, or B) visit 1 page to see that same info. :? IMHO the "A" option is even more work than the option "B".

Because this is something they'll be told to do everytime they run the program as opposed to remembering to check your approved driver page... The point here is to make the reviewers think about what they are doing, and making them take a decision. Do we follow the rules of Futuremark that we are presented with every time we're using a new set of drivers (ie. check if they're approved, if not use older drivers), or ignore the rules and just show our readers a result which might very well have been tampered with...

It will also (as I've said previously) let more people know of the problem with un-approved optimized drivers.

Imo, this will lead to the readers to put more pressure on the reviewers to use 3DMark the proper way, which in turn might turn the pressure to Nvidia for them to remove said optimizations. But this will only happen if you make the public see that 3DMark is a worthwile benchmark to use... (A bit catch 22, but still)

]

Herein lies the real problem imo, as the reviewers feel, imo, it's too much work to test with 2 different driver versions, when only one set is partially approved...

"Why bother to test with an older driver set, when the result still won't be compareable to cards from another IHV?" That is what I think many reviewers feel about it, even though it's only the PS2.0 test that is incomparable.

Partially? IMO that sounds like only a small fraction is approved. We are talking about 1 theoretical test (out of 14 default performance tests ). Besides, most sites use the 3DMark score in their reviews. That result is comparable.

(My bolding)

Exactly my point! Which was why I suggested a re-wording of the text concerning the 52.16 drivers. And doing that should be very easy to do....

]

To be very blunt, what would be so hard about it?

As for if it would make a difference, yes I think it would. You would be told every time what drivers are approved, and it would be in an "in your face" kind of way it'd be hard to ignore.

You make it sound so easy. In all honesty, I am not a coder so I can not say how hard it would be to implement. However, I know that it would require a lot of extra work. We would need to not only change the software itself, but we would also need to cook up the online service too. I am not sure how the program (3DMark03) was built, but I have a hunch that if we would add anything new to it (like the auto-updater) we would need to open up more than 1 piece of code in order to get it work properly.

Yes, I make it sound easy doesn't I...
And a bit depending on which way you decide to tacle it, it can be easy...
Sending a call to the standard web browser to download a file at a give url, and then decode that file... And then the driver version checking (which might be some work)...

But does it hurt to ask your coders how much work it would be to implement it?

]

Yes, you can only do so much up to a point, but I think most of this board (who still cares about 3DMark) feel that that point is far from reached, and won't be reached until it's obviously blatant to any person running 3DMark..

Hmm, I still believe that we have done quite a lot. We have enforced our guidelines, started the Approved Drivers testing etc. We have informed the media about this, posted it clearly on our website, made notifications in the ORB about it etc. I doubt that many who use 3DMark03 more than once have missed it by now. Just submit 1 result to the ORB, and you can see & read about it. Anyway, we will of course keep working on this, and hopefully all for the better.

It's good that you are working on this (anything else would surprise me). But just make sure that the public is aware of the fact that you are doing it... Maybe with some news items on your frontpage & such?

]

On another note, it'll be interesting to see what "special" thing you're talking about that might be released this week...
If it's another patch, it'll hopefully remove all optimizations in at least one driver revision so we won't be in the position we are with 52.16

Sorry, no patch. It isn't something that "special", and is targetted foremost at the online & offline media.

Ah, well... It was possible to hope... But it'll still be interestin to see what it is, since it's mentioned managed to stop Dig from screaming bloody murder anymore...

 

[digitalwanderer]

Ah, well... It was possible to hope... But it'll still be interestin to see what it is, since it's mentioned managed to stop Dig from screaming bloody murder anymore...

It's pretty easy to figure out, if you don't know just go back and read the last four or five threads on the subject and it'll be plain as day. (Sayeth the Dig with a mischevious twinkle in his eyes thinking of just how many pages of posts that would be. I never mind reading long threads that I participate in since I only read a bit of 'em at a time, but coming to a 10+ page thread and just reading it cold can be a real ball-crusher! )
Basically I figure if they're working on putting something out soon then yelling at 'em to hurry up would be kind of counter-productive, I'm waiting with interest to see what the reaction is when they do what they're going to do. (And I'm curious about how they actually plan to do it, but it's another time for me practicing me patience....which is probably good since I need practice at it. )

 

[MrGaribaldi]

Ah, well... It was possible to hope... But it'll still be interestin to see what it is, since it's mentioned managed to stop Dig from screaming bloody murder anymore...

It's pretty easy to figure out, if you don't know just go back and read the last four or five threads on the subject and it'll be plain as day. (Sayeth the Dig with a mischevious twinkle in his eyes thinking of just how many pages of posts that would be. I never mind reading long threads that I participate in since I only read a bit of 'em at a time, but coming to a 10+ page thread and just reading it cold can be a real ball-crusher! )

I think the small text said it all when it comes to checking old threads.........

Basically I figure if they're working on putting something out soon then yelling at 'em to hurry up would be kind of counter-productive, I'm waiting with interest to see what the reaction is when they do what they're going to do. (And I'm curious about how they actually plan to do it, but it's another time for me practicing me patience....which is probably good since I need practice at it. )

Good points... Especially the one about need practice in patience

 

[BetrayerX]

.... or one that wont run if the drivers are not approved.

that's the last thing we need. if a 3dmark did not run on "unappoved" drivers, we'd (and futuremark, of course) hear no end to the "you are showing bias twords <company X>!!" from a certain camp. furthermore, it would be impossible for anyone to confirm (or disprove) accusations of cheating.

perhaps a warning message when 3dmark is launched (drivers not approved! press "ok" to continue) and/or blatently labeling results as being obtained using unapproved drivers would be in order; but having the program fail to run would be worse imo than the situation that exists now
c:

I disagree. I think it would be a bit extreme, but at least they would be showing an effort to enforce their programs rules.

Not necesarilly. Labeling those scores as "non-approved" coming right out of the bat is a good policy. Yet they can't forbid someone to use the test because of that. There are some people out there that care for having the highest score (as lame as that sounds, it's true), so everything is valid for them.

 

[BetrayerX]

]

don't forget guys there are other synthetic tests out there some much better In My Personal Opinion

D3D RightMark, and ShaderMark for example

Just out of curiosity, technically how are they much better and how can you be sure that there aren't any app specific optimizations in any drivers for those apps? Just for my Personal Interest.

Just to point that NV boards don't even finish ShaderMark tests, shows that those boards have limitiation (hardware and/or software, but since the driver team is so good you can guess the hardware is lacking in the shader dept.)

 

[just me]

NOTE: This project has not been run with Futuremark approved drivers, or the status of the drivers can not be determined. The accuracy of the results can not be guaranteed. For more information on how to get a valid and comparable result, please refer to this page.


That's what you see online now in 3DM03 for non-WHQL/non-Approved drivers like the 98SE/Me drivers.

Been there since Thursday, that I know of. "this page", in the text, is a link to 'Approved'/WHQL drivers for 2000/XP.

 

[digitalwanderer]

NOTE: This project has not been run with Futuremark approved drivers, or the status of the drivers can not be determined. The accuracy of the results can not be guaranteed. For more information on how to get a valid and comparable result, please refer to this page.


That's what you see online now in 3DM03 for non-WHQL/non-Approved drivers like the 98SE/Me drivers.

Been there since Thursday, that I know of. "this page", in the text, is a link to 'Approved'/WHQL drivers for 2000/XP.

At least it's red.

 

[Martin Eddy]

Yeah The Dig certainly likes his colours.

 

[AzBat]

]

Worm you're not that dense are you? The whole point is that there shouldn't be a need to go to your Approved Drivers page every time somebody uses your benchmark(I would, but that's just me). The update in the software is there to officially notify them they may or may not be using approved drivers and to allow them the opportunity to have the 3DMark03 software to check via the Internet for the latest list. This wouldn't require the actual user to open up their browser and go to the page itself. Remember they have already started the software and are preparing to benchmark. So, what's so hard for the software to check the net provided they have a connection? They're already using the ORB right?

Ok, it seems that I wasn't clear enough in my comment, sorry for that. I was merely refering to IF the user doesn't use the auto-updater, but rather downloads the "approved drivers list" and installs it manually. If the user has a net connection to his test system (which is what I have been told that very few have. I might be wrong though.) then the auto-updater might be useful for the approved driver listing. But that is just if. Am I making more sense now? I mean, downloading a patch manually, installing it manually and then starting the benchmark to see if the drivers are approved or not, is IMHO much more hassle than simply heading to one webpage to see the same thing.

OK, I agree that typing your web site address into a browser to download an "approved drivers list" is more hassle than just going the official web page that has the list. So I don't think that's a legitimate suggestion. However, if you provide a button in 3DMark to download the list without having to quit the program and open the site using a web browser, then that is something that I think that is much more useful(provided they have a Internet connection). If they don't they would either have to make a connection or go to the "approved drivers list" page manually when outside of 3DMark.

]Besides, I personally find it a tad odd that so few reviewers include compare URL's to their reviews. I mean, they usually have the Pro version, which means they can upload unlimited amounts of results, so why not use it? I know users would love to see that too as they can view the result in detail without even having to register to the ORB. So you could say that most reviewers do not use the ORB, or if they do, they don't talk about it in their reviews.

I agree that for reviewers to not use ORB or compare URL's is strange and actually quite disappointing. It's actually something an end-user could actually use to determine if a new card is worth an upgrade. I believe the reasons why it's not used to due to probably three factors: 1) no network connection on test computer or temporarily disconnected during test 2) don't like the idea of the public(FutureMark, end-users, IHVs and competitors) having so much info about their computer test set-ups 3) it sends visitors off their site to FutureMark and therefore they lose ad revenue.

]But the point is that no matter what auto-updaters or whatnots we would do, it is still up to the reviewer what he is going to do. If someone has set his mind on NOT to use our approved drivers, then no notifications or like will do any good.

Understood, but by providing them with a "nag screen" for not using approved drivers eventually you should be able curb its use. "Nag screens" do seem to work in the shareware business. Maybe you should also quit giving the Pro version away free to those sites that don't use the approved drivers? Might have backlash, but at least you wouldn't be subsidizing and sanctioning the sites that don't abide by the EULA.

]As I posted earlier, we will try to improve the flow of information, but at the end of the day, it is up to the reviewer/user what he uses. And before anyone jumps to any wrong conclusions, this does not mean that we would stop working around this! As said, we will continue to work on the ORB to inform users about the drivers, we will start sending out more information to the press (DW ) and we will work on other solutions.

Looks like you been doing more from the messages above. I'm going to check your site and forums to see if anything official was announced. Hopefully you did.

Tommy McClain

 

[just me]

Looks like you been doing more from the messages above. I'm going to check your site and forums to see if anything official was announced. Hopefully you did.

If you do find anything on the FM Forums about what I previously posted > lemme know. The only way I knew about the above message was to upload a new score. I'm on 98SE & there are no WHQL drivers for 98/Me so I got the message.

I havn't seen anything on FM's site about it tho', except what I posted there informing others > just like I did here.

I've always wondered why B3D gets posts by worm & other FM admin/staff, but they seldom post on FM. I'd think they'd want FM members to be the 1st to know ..... :?

8)

 

[X*Devourer*X]

This may have been mentioned before but here is an idea for making sure reviewers/sites use approved drivers.

#1. Have an approved site review list with all sites on the list having signed an agreement stating to gain access to FM's approved reviews list they will use drivers only approved by FM for there reviews. Have them add a (approved review by FM) to every review which uses 3dMark03 for any reviews. This will not only help with cheating but it will also help with people unaware of cheating drivers that can be used.

#2. Have separate 3dMark03 programs for said review sites that will allow them to post there results on FM's site in a special FM 3dMark03 review results page. All other people downloading 3dMark03 benchmarks are forced to post there results in a common area with (in huge letters stamped across there results) non approved drivers if they are in fact not approved or just simply refuse to allow non approved driver results to be posted at all.

Just my 2 cents on options to help prevent reviewers from using cheat drivers & I apologize if this idea has been brought up already.

 

[Nick[FM]]

Looks like you been doing more from the messages above. I'm going to check your site and forums to see if anything official was announced. Hopefully you did.

Hmmm, there's nothing to "announce" exactly. Our users use the ORB and other features of ours on a daily basis, so they are very aware of what is happening. If we add some new information to the ORB, or add some copy here and there, we don't make announcements of them. The reason why I am posting it here is because of this thread and where it has headed. If I would see any such discussion anywhere on our site/any other site, I sure would do the same thing.

 

[digitalwanderer]

]Hmmm, there's nothing to "announce" exactly. Our users use the ORB and other features of ours on a daily basis, so they are very aware of what is happening. If we add some new information to the ORB, or add some copy here and there, we don't make announcements of them. The reason why I am posting it here is because of this thread and where it has headed. If I would see any such discussion anywhere on our site/any other site, I sure would do the same thing.

Well I like it!

Have you received any feedback from it yet? I'm mainly interested in any IHV feedback, but any feedback news would be interesting.

I was thinking of writing up a ditty about it, would you mind? :|

 

[jb]

so with those of us stuck at work and no access to the ORB....do tell?

BTW Digi, hows the painting coming along

 

[digitalwanderer]

BTW Digi, hows the painting coming along

I fixed the bed, got all the stuff out, and I'm going to go and scrub the walls down now...thanks for prodding me out of my chair.

 

[jb]

BTW Digi, hows the painting coming along

I fixed the bed, got all the stuff out, and I'm going to go and scrub the walls down now...thanks for prodding me out of my chair.

Nooo you were suppose to tell me what happen on the orb as I can not hit that from work.....

 

[Nick[FM]]

Have you received any feedback from it yet? I'm mainly interested in any IHV feedback, but any feedback news would be interesting.

I hope we are talking about the same thing here.. Anyway, the first issue will be sent out asap, but due to some travelling (US-Europe) it has been delayed a bit. This week is looking good though!

I was thinking of writing up a ditty about it, would you mind? :|

As long as we are talking about the same thing.. I am not sure what good an editorial (or something) really would do, but that's really up to you!

so with those of us stuck at work and no access to the ORB....do tell?

I think the update from the ORB was already posted here in red? We added a notice to the projects which aren't using approved drivers.

 

[jb]

I think the update from the ORB was already posted here in red? We added a notice to the projects which aren't using approved drivers.

Doh!!!! Sorry Thanks!

 

[Nick[FM]]

I think the update from the ORB was already posted here in red? We added a notice to the projects which aren't using approved drivers.

Doh!!!! Sorry Thanks!

No problem!

 

[digitalwanderer]

Well it looks like Hanners is thinking that his suggestion will be it....I still want the screen to flash red and wailing alarms to go off if you try and run 3dm2k3 with un-approved drivers though.