New 3DMark03 Patch 330

nelg said:
I just feel that if it was mandatory it might discourage some sites from using it.

I don't think so. Sites that are currently use it and find value in it will want to continue using it. Those that don't, won't. For example, considering Kyle's opinion of 3DMark, I doubt [H]ardOCP would become a member. Though Brent could possibly argue for it.

nelg said:
That would not be in Futuremarks interest. I am not suggesting a level of involvment like that of a beta member. Just a way to keep sites using the benchmark correctly.

I don't know the extent of the current beta program that B3D is now a part of, but if it's anything like it was when I beta testing 3DMark99 through 3DMark2001, then I would say there's really no high level involvement. They never required anything of me other than sending my comments to them when I tested the beta builds. With that said I don't see why review/tech tech sites would NOT want to be a beta member.

nelg said:
If there was a certification process maybe some sites would be cautious about doing such things for if they did they would loose there certification. As a layman in this area I personnally would feel better if a site was labeled "CERTIFIED by FUTUREMARK".

Agreed.

Tommy McClain
 
worm[Futuremark said:
]
AzBat said:
Maybe even go a step further by training and certifying those sites to make sure they understand the benchmark, it's results
Actually.. I have been planning / throwing some ideas around (in my head only so far) to organize a "Benchmarking Bootcamp - Online" or whatever you want to call it. ;)

Hehe. Great idea! I loved the first and only Benchmark Bootcamp in San Francisco. I would love to see it continue, even if online. Would we get free t-shirts and sweats again? ;)

worm[Futuremark said:
]It could either be once a month, an ongoing thing, or then once a year (bigger happening). At least something where online, and offline, media could come to discuss about 3DMark & PCMark, listen to "lectures" etc. about our products. To get the best possible guidelines how our products are to be used / not be used, and maybe some instant help if there are some issues / problems.

Agreed. This was one of the reasons the ZDlabs Benchmark Tech Forum was good. Since their demise, there hasn't been anything that has tried reproducing this kind of industry event. If Futuremark would do this, then I really think it would go a long ways to help legitmize 3DMark as industry leading benchmark. I think a major event every year for all beta members would be good for introducing new benchmarks. Then once a month have some kind of online get-together for just the press/online community. You could also send out weekly tech bulletins via email or have a mailing list for just the press/online community. There's no need for NVIDIA, ATI, etc to know about the issues we're discussing until it's made public.

worm[Futuremark said:
]"Certified 3DMark Site" or something similar would be a good thingâ„¢. :D The idea is nice! Maybe some test in order to get it? Hehe..

Code:
Question1: How many Game Tests are there in A) 3DMark99 B) 3DMark2000 C) 3DMark2001 and D) 3DMark03
Question2: Using what formula is the CPU score calculated in 3DMark03?
Question3: What did the trolls in GT3 (3DMark03) do when the heroine interrupted them?
Question4: How many separate speeders were there in 3DMark99 GT1?
Question5: Who made the music for A) 3DMark99 B) 3DMark03
;) Hehe..

A think a test would be beneficial. Though I'm not sure of the kind of questions. Should they be similar to ones you gave or more technical? A more technical test could help reviewers in their understanding of the benchmark, which in turn help their visitors.

worm[Futuremark said:
]But seriously, I will post more about this if it will ever become a reality. Any ideas & suggestions concerning it, please email me! It's better that way as I might miss some good pointers if they are posted in some thread somewhere. Of course posting here is ok too, but there's always a chance that it gets unnoticed.

OK. I'll send you an email if I come up with anything else. ;)

Tommy McClain
 
AzBat said:
I don't know the extent of the current beta program that B3D is now a part of, but if it's anything like it was when I beta testing 3DMark99 through 3DMark2001, then I would say there's really no high level involvement. They never required anything of me other than sending my comments to them when I tested the beta builds. With that said I don't see why review/tech tech sites would NOT want to be a beta member.

Some sites are run by part-time by hobbyist. I could see them not wanting to expend the time to be “certifiedâ€￾. Even if a site did not want to invest the time or energy in the process I don’t think that it should be excluded from using the benchmark. As far as what kind of involvement beta members like B3D and ET have compared to your previous maybe Dave could elaborate ?
 
nelg said:
Some sites are run by part-time by hobbyist. I could see them not wanting to expend the time to be “certifiedâ€￾. Even if a site did not want to invest the time or energy in the process I don’t think that it should be excluded from using the benchmark.

I was a part-time hobbyist when I ran Dimension3D so many years ago. ;) With that said, I would want to become "certified" if it meant I could publish 3DMark benchmarks. I'm not saying that a lot of time and energy should be required to be certified. But then again, after all of these of kind problems, I'm going to want to make sure that everybody that does use 3DMark results on a web site or magazine review understands what it is and what it is not. They have that responsibility to not only their visitors or readers, but also to Futuremark for making sure it's represented fairly. 3DMark can be a very valuable tool, but not if they're not using it right or don't understand it. If a site is not willing to do what's necessary become certified, then who is to say they spent the proper amount of time and energy in their reviews? Any site that would complain about not wanting to be certified, is a site I wouldn't visit or trust.

Tommy McClain
 
I'm w/nelg on this one. The burden is on FM to ensure the bench stays viable, not review sites. "Mandatory Beta Membership" for a free downloadable bench? Only way FM could do that is a clause saying it can't be used for commercial purposes w/o membership. That will kill its usage.

I applaude the idea of an open source for training & usage of the Marks > shoulda been in place a long time ago. I applaude that some sites do police Mark usage ... but I have to ask: If B3D/ET wouldn't have found the cheats > would FM have ever found them?

FM makes $$$ off of their products. The burden of legitimacy is upon them. Creating a police force out of paying beta members to ensure the bench isn't cheated is something I'm sure FM would gladly want > cuts down their workload.

FM "Certified" sites? NO! FM "Certified" drivers > YES! I like FM & its benches, but I have to wonder how many other drivers 'got by them' & why they didn't catch these cheats & had to be informed by others. I thought after the last driver fiasco they said they were going to check & disable all subsequent cheating drivers. Then how did this 'driver' get by them & Dave found it?

I support you FM, but it's your burden to ensure compliance of the drivers w/the bench > not 3rd parties, IMO. Making those 3rd parties pay for membership & police your proggie to use it? :rolleyes:

.02,
 
just me said:
I'm w/nelg on this one. The burden is on FM to ensure the bench stays viable, not review sites. "Mandatory Beta Membership" for a free downloadable bench? Only way FM could do that is a clause saying it can't be used for commercial purposes w/o membership. That will kill its usage.

I applaude the idea of an open source for training & usage of the Marks > shoulda been in place a long time ago. I applaude that some sites do police Mark usage ... but I have to ask: If B3D/ET wouldn't have found the cheats > would FM have ever found them?

FM makes $$$ off of their products. The burden of legitimacy is upon them. Creating a police force out of paying beta members to ensure the bench isn't cheated is something I'm sure FM would gladly want > cuts down their workload.

FM "Certified" sites? NO! FM "Certified" drivers > YES! I like FM & its benches, but I have to wonder how many other drivers 'got by them' & why they didn't catch these cheats & had to be informed by others. I thought after the last driver fiasco they said they were going to check & disable all subsequent cheating drivers. Then how did this 'driver' get by them & Dave found it?

I support you FM, but it's your burden to ensure compliance of the drivers w/the bench > not 3rd parties, IMO. Making those 3rd parties pay for membership & police your proggie to use it? :rolleyes:

.02,

Nowhere did I say that the press should be the only ones policing the benchmark. Yes, Futuremark has a responsibility to police the benchmarks. They do have more tools than we do. But to think that Futuremark is big enough to find all cheats themselves is ridiculous. I mean, the press is the ones using the benchmark day in and day out, why not take advantage of that and help them out? Hell, they already put tools in the benchmark to do just that. You think they're putting it in there for the hell of it? :D

Making it mandatory for press and the online community would be just an extension of their license agreement. There is already stipulations in there for people publicly posting benchmark results. Individual use should continue to be free and not require beta membership. That hasn't killed it usage so far.

Also, nowhere did I say that the press should have to pay to become members of the beta program or be certified. I think it should be free. It wasn't when I was beta testing. If they're now requiring the press to pay, then God help them.

Tommy McClain
 
Nowhere did I say that the press should be the only ones policing the benchmark.

Me either. I just don't think they should be required to be police to use it.

Yes, Futuremark has a responsibility to police the benchmarks. They do have more tools than we do. But to think that Futuremark is big enough to find all cheats themselves is ridiculous.

Ridiculous? Not in my eyes. It is their 'cash cow' & if they need more ppl > hire 'em. Asking for help is fine & those that volunteer to do so should be commended, but mandatory membership for usage in lieu of the license agreement as it stands > no!

I mean, the press is the ones using the benchmark day in and day out, why not take advantage of that and help them out? Hell, they already put tools in the benchmark to do just that. You think they're putting it in there for the hell of it? :D

Day in & day out? Geez, we only get WHQL drivers every 6weeks & NV went (what) 4months on the last one? Yeah the tools are also available to FM to use too. "Lead by example".

Making it mandatory for press and the online community would be just an extension of their license agreement. There is already stipulations in there for people publicly posting benchmark results.

From the License Agreement:

License Agreement

3DMark03 - S O F T W A R E L I C E N C E A G R E E M E N T

...

4 Restrictions

...

It is expressly prohibited to collect and publish or cause to be published any publicly accessible database of information obtained by the use of the Software. This prohibition is waived for independent press, who may publish a database of such information for purposes of reporting PC performances to their readership.


What 'stipulations'? What 'agreement' specific to review sites? The 'independant press' can publish '03 results (Free, Pro or Corp version) w/o any 'stipulations' or as FM says: "prohibition".

Individual use should continue to be free and not require beta membership. That hasn't killed it usage so far.

Also, nowhere did I say that the press should have to pay to become members of the beta program or be certified. I think it should be free. It wasn't when I was beta testing. If they're now requiring the press to pay, then God help them.

Agreed. 8)
 
just me said:
Nowhere did I say that the press should be the only ones policing the benchmark.

Me either. I just don't think they should be required to be police to use it.

Neither do I. Certification or being a member of the beta team doesn't mean you are required to police it.

just me said:
Yes, Futuremark has a responsibility to police the benchmarks. They do have more tools than we do. But to think that Futuremark is big enough to find all cheats themselves is ridiculous.

Ridiculous? Not in my eyes. It is their 'cash cow' & if they need more ppl > hire 'em.

That's an insane idea. How many driver developers work for ATI, NVIDIA, etc? How many 3Dmark developers work for Futuremark? I sincerely would like to know. Somehow I don't see Futuremark being able to hire as many people that would be enough to offset the amount that IHVs have.

just me said:
Asking for help is fine & those that volunteer to do so should be commended,

Agreed.

just me said:
but mandatory membership for usage in lieu of the license agreement as it stands > no!

I don't understand why you're so against this idea? Are you or have you ever done benchmarking for a web site or other publication? If you're a end-user I would expect you to welcome the idea.

just me said:
I mean, the press is the ones using the benchmark day in and day out, why not take advantage of that and help them out? Hell, they already put tools in the benchmark to do just that. You think they're putting it in there for the hell of it? :D

Day in & day out? Geez, we only get WHQL drivers every 6weeks & NV went (what) 4months on the last one? Yeah the tools are also available to FM to use too. "Lead by example".

Umm. Haven't Futuremark done just that with the PDF and patch?

just me said:
What 'stipulations'? What 'agreement' specific to review sites? The 'independant press' can publish '03 results (Free, Pro or Corp version) w/o any 'stipulations' or as FM says: "prohibition".

Could you email me a copy of the license agreement? I remember seeing stipulations for publishing benchmark results. You were required to publish the test sytem specs and other details regarding the test procedures. It's possible I have it confused with the 3DWinBench license agreement, but I could have sworn Futuremark had similar requirements.

Tommy McClain
 
but mandatory membership for usage in lieu of the license agreement as it stands > no!

I don't understand why you're so against this idea? Are you or have you ever done benchmarking for a web site or other publication? If you're a end-user I would expect you to welcome the idea.

I value the 'independant press' & the idea that 'independant' implies impartial. 'Membership' implies otherwise, to me.

Could you email me a copy of the license agreement? I remember seeing stipulations for publishing benchmark results. You were required to publish the test sytem specs and other details regarding the test procedures. It's possible I have it confused with the 3DWinBench license agreement, but I could have sworn Futuremark had similar requirements.

It's in the "Help" file included w/'03. Last page of the HTML document. It does have certain 'stipulations' but, they are "Subject to other restrictions stipulated in this Agreement, " Since the preceeding section applying to "Resirictions" says "This prohibition is waived for independent press", the section you remember doesn't apply to 'independant press' > they aren't "Subject to" the "Restrictions" listed & are specifically singled out as not 'prohibited'.

If you don't see it that way, look at this from that section:

VII. State that all products used to obtain the Result were shipping versions available to the general public.

Every Preview of a 5900U is violating that & that clause would kill all ES/BETA benchs w/'03. ;)

If you don't have '03, I will email you the file.
 
That's an insane idea. How many driver developers work for ATI, NVIDIA, etc? How many 3Dmark developers work for Futuremark? I sincerely would like to know. Somehow I don't see Futuremark being able to hire as many people that would be enough to offset the amount that IHVs have.
From the 3DMark03 credits list:
Code:
Producer
Patric Ojala

Programming
Petri Häkkinen
Mikko Kallinen

Graphics
Mikael Bowellan
Eetu Martola
Juha Pinola
Janne Pulkkinen
and 2D by Pertti Kainulainen

Additional Programming
Jarkko Lempiäinen
Yongjun Zhang

Additional Graphics
Petri Häkkinen
Olli Hänninen-Pelz

Sound Effects
Markus Kaarlonen
Nicklas Renqvist
 
just me said:
Could you email me a copy of the license agreement? I remember seeing stipulations for publishing benchmark results. You were required to publish the test sytem specs and other details regarding the test procedures. It's possible I have it confused with the 3DWinBench license agreement, but I could have sworn Futuremark had similar requirements.

It's in the "Help" file included w/'03. Last page of the HTML document. It does have certain 'stipulations' but, they are "Subject to other restrictions stipulated in this Agreement, " Since the preceeding section applying to "Resirictions" says "This prohibition is waived for independent press", the section you remember doesn't apply to 'independant press' > they aren't "Subject to" the "Restrictions" listed & are specifically singled out as not 'prohibited'.

If you don't see it that way, look at this from that section:

VII. State that all products used to obtain the Result were shipping versions available to the general public.

Every Preview of a 5900U is violating that & that clause would kill all ES/BETA benchs w/'03. ;)

Here's the stipulations I was talking about...

Code:
11 Usage of results and Trademarks 

Subject to other restrictions stipulated in this Agreement, Futuremark hereby grants You the right to publish, except in any country where a third party claims during the term of this Agreement that such publication infringes that party's proprietary rights, information ("Result") obtained by You from Your use of the Software, provided that with the publication of each Result You: 

I. Identify the Licensor, the name and the version number of the benchmark Software used and Futuremark as the source of the software (i.e., "3DMark®03 Pro by Futuremark®"); 

II. State that the test was performed without independent verification by Futuremark, and that Futuremark makes no representations or warranties as to the result of the test; 

III. Follow proper trademark usage and acknowledge the Futuremark's trademark rights according to Futuremark's policy (e.g., "Product X achieved Y 3DMarks® on 3DMark®03 Pro. 3DMark® is a registered trademark of Futuremark® Corporation"); 

IV. Identify the specific result(s) being reported. (e.g., "3DMark®03 score of 4857, Vertex Shader test result 23.7 fps"); 

V. Identify the settings in 3DMark® that have been used to obtain the Result; 

VI. Identify the exact CPU name, type and speed, amount of L1 Cache and L2 Cache memory, number of processors, amount, type and speed of Physical Memory (RAM), the 3D accelerator manufacturer and brand name, 3D accelerator product and model name, 3D accelerator driver version, amount and type of video memory on display card, AGP bus type and speed, Operating System name, version and possible installed Service Packs, DirectX® version and any other special conditions or settings of the PC used for the test that could affect the 3DMark®03 results, like increased 3D accelerator core and/or memory speed; 

VII. State that all products used to obtain the Result were shipping versions available to the general public. 

VIII. Upon request from Futuremark, submit a sample of the publication in which the Result was published.

To me it is very clear that "independent press"(web sites, etc) are still required to follow the 8 points outlined above when posting benchmark results.

Anyway, when I said the following...

AzBat said:
Making it mandatory for press and the online community would be just an extension of their license agreement. There is already stipulations in there for people publicly posting benchmark results.

I guess I should have said COULD instead of would. Meaning that they would need to update their license agreement in order to require sites and press to be certified before publishing results.

Considering the amount work web sites and other press are required to do when posting benchmark results, I don't consider it much of a stretch for them to get certification from Futuremark to allow them to do so.

Tommy McClain
 
Well, if for some reason everyone pulls out of the beta program and no one buys the Pro version, FM can still make money by adopting Rambus' model and sue pretty much every hardware website that's used 3DMark for breaking the license.
 
AzBat said:
That's an insane idea. How many driver developers work for ATI, NVIDIA, etc? How many 3Dmark developers work for Futuremark? I sincerely would like to know. Somehow I don't see Futuremark being able to hire as many people that would be enough to offset the amount that IHVs have.
Undoubtedly nvidia/ATI have more resources than Futuremark. However, if Futuremark can induce enough fear that cheaters will be caught then it will become unprofitable to try to cheat on 3dmark - and a relatively limited amount of work will be needed to police it as long as they keep their eyes on the ball.

For this to happen, though, cheating has to cost something to the cheating company's bottom line.
 
The way I see it, hardware sites should run the 3DMark2k3 benchmarks twice, once using 320 and then again using 330. This will show consumers if 1st) the hardware is actually good 2nd) the drivers actually make thhe hardware good.

I did a few tests here using 43.45, 44.03 and 23.11 Drivers and was pleasantly suprised by the results. Nvidia are definitely cheating using the 43.45 and 44.03 drivers(and one has to wonder what other drivers too), but with the 23.11 driver the results were the same on 3DMark2k3 320 and 330 which goes to show that older drivers were not tweaked by Nvidia. What was a shocker though was that the older 23.11 drivers actually have a higher score than the 43.45 drivers running 3DMark2k3 330 build which means that those drivers were actually alot better than the 43.45 drivers although they were not optimised for the benchmark.

In my opinion, Hardware sites(and any worthy site) should continue using 3DMark2k3 with the latest build but they should also make a comparision with the older build too, just to show if Nvidia and ATI intend to continue cheating.

US
 
but with the 23.11 driver the results were the same on 3DMark2k3 320 and 330 which goes to show that older drivers were not tweaked by Nvidia

Was 3dmark03 out when the 23.11 drivers were released?
 
Digit-life has some animated gif's show the difference between 3.20 and 3.30. Also some strange comments like...

"The reputation of the 3DMark03 is spoiled by the scandal and by the fact that the test developers preferred to exhibit arrogance to normal collaboration with all chip makers. "

:oops:
 
Back
Top