Ops,"I" did it again. Why does Futuremark still ke

[Mariner]

Cow-Towing

I know what you mean here (spelt kowtow, by the way), but the way you have written it brings a peculiar image to mind!

 

[Reverend]

(I am a little Irritated right now.. so please bear that in mind)

99% of your posts show that you're in a irritated state. The reason why you're irritated is obvious but I think your health will improve if you calm down a bit when participating in a public forum such this site's (which btw should also help improve this site's forum quality).

 

[AzBat]

ouch.

Actually, in thinking a bit more about the conflict of interest inherent in the business model, it strkes me there are actually two major conflicts of interest there.

The first is that fact that while the assumption of the IHV partners is that they are paying for software which can be used as a marketing tool to portray their products in a positive light, the public is thinking they are using software which provides a level playing field for an ojective evaluation of the features supported by differing 3d hardware in a comparative context.

The second is that you have competing IHVs paying dues, but expecting entirely different things from the software. For instance, nVidia pays x dollars and tells FM to "go this way" with its software, and ATi pays FM x dollars and tells them to "go that way" with its software. So what does FM do? If they go "this way" they risk alienating ATi and having ATi quit the program, but if they go "that way" with it they risk alienating nVidia and having nVidia quit the program (which has already happened.) Last, if they take the middle route and split the difference between them then they risk alienating both of the IHVs and possibly having both of them quit the program.

Pre-R300 I'm guessing that conflict of interest between IHVs was not that much of a problem since nVidia had a pretty good lead at the high end. But now that ATi is emerging as dominant, I can see several bones of contention developing along the lines that prompted nVidia to quit last year. As the competition intensifies between the IHVs my thought is that the situation isn't likely to subside, but rather to grow more acrid.

The funny thing about companies is when they shell out money for programs like this they naturally expect some tangible benefit in the way of goods and services in return. If they don't perceive they are getting such benefits they lose the motivation to pay for the privilege of participating. Looks like a thorny problem for FM going forward.

I think the best solution therefore would be for them to change their business model and deal with the IHVs on a voluntary basis through NDAs and drop them from the partnership dues-paying rolls. They could maintain their partnership with oems like Dell (and other entities) for the purposes of sales and distribution, and concentrate on creating benchmark software for sale to the general public, primarily to serve the general public. If an IHV wanted to see its hardware represented in a forthcoming FM benchmark it could supply whatever help it wanted to FM, sans payment, and FM could derive its income from its non-IHV partners and sales of its software to the general public. That way FM could eliminate these potentially serious conflicts of interest entirely. If they don't do something like this I can see these conflicts eventually tearing the company apart. Just my opinion, of course.

Great posts Walt. I would have to agree with everything you said. In fact, if I were still writing benchmark articles for JPA I would have written the same thing. In fact, why don't you forward your comments to my former boss which still works for Jon at Jon Peddie Research? Her email address is kathleen@jonpeddie.com. Tell her I sent you and maybe she'll include them in next week's TechWatch Report.

Anyway, FM really needs a reality check. During the time I was doing beta testing for 3DMark since before the creation of Futuremark, there was no cost to IHVs to be part of the program that I was ever aware of. I understand that they need capital to pay for development costs, but they should seriously reconsider their business model. I remember the day that Nathan Harley came to San Francisco to meet with me, my boss and Jon Peddie about their plans for Futuremark. I was excited and genuinely liked the company's new ideas. I don't know when they started making IHVs pay to be part of the program, but I would have to say that probably started their demise. It's pretty bad when you fire your whole Toronto(marketing) office which included a founder, but to also have another founder leave to work on MP2 and yet another to leave after this fiasco. I want everybody to know that I was once approached by Nathan and AJ to work on the 3DMark program. Things didn't work out. I guess it was bad timing, but I'm kinda glad it didn't work out because I don't think I could have stayed with the company knowing what's happened recently. I'm even ashamed that my name is still listed in the 3DMark credits. In my eyes, FM has only 2 things they can do to regain my trust...

1) Quit accepting money from IHVs to be part of the 3DMark BETA program. Make your money from end-users, OEM bundles, selling of benchmark data, support services and custom demo services. If that's not enough, then think of other ways to make money, but do not continue the conflict of interest by requesting thousands of dollars from IHVs.

2) Stand-by your original stance posted in the white-paper. This means actively searching for cheats and optimizations. If any are found then they need to be rejected publicly and results removed from the database. In other words, grow a spine and don't let the IHVs push you around. The point is that all hardware should run the exact same code. If you want to provide additional tests that are not included the score, but also allow shader replacements, then that's fine, but DO NOT ALLOW application specific be used solely for increasing scores or performance.

Tommy McClain

 

[Pete]

I don't see the conflict in your first point, Walt. The public could care less about the black box that is game code and hardware features and drivers--they just want a good-looking game at playable framerates. It's the developer's job to deliver that, multiple hardware standards or not. It's an uneven "playing field" to begin with, if you consider differing feature sets or hardware optimizations to be uneven. I consider it a reality that must be dealt with. FM can surely devise benchmarks that reflect well on both ATi's and nV's current hardware, as game developers are forced to do.

I agree with your second point. It seems antithetical to 3DMark's goal of an impartial benchmark to allow IHVs pay to be Beta members. It's almost like blackmail--FM starts charging after they find themselves the only game in town in terms of one-click 3D evaluation,* and IHVs are forced to pay a relatively small fee in relation to their overall profits (consider it part of their marketing budget) to stay competitive in the eyes of the always-important VARs (like Dell). From this perspective, I can agree with nVidia's dislike of 3DM03, but I still can't excuse the actions they took to achieve their goal of discrediting 3DM.

In the end, 3DM seems to still be a potentially useful tool if you know how to use and interpret it--much like any other benchmark, and moreso given the ever-widening field of IHV-forced "optimizations." Just as you can't extrapolate Quake 3 performance to Splinter Cell or Tiger Woods or even another FPS based on the Q3 engine, you shouldn't expect a catch-all benchmark written by a single group of developers. If reviewers and companies want to present their less-diligent readers/consumers with an easy-to-digest number, they're better off with an amalgamated benchmark like ExtremeTech's GameGauge that takes into account performance of a variety of game categories coded by a variety of developers. They're still bound by developers including a repeatable, simple benchmark mode, though, and that's one area where FM excels.


* Quick, someone patent this before Amazon does!

 

[Solomon]

Patric,

I just read your post in your personal forum here at B3d, and I've read your comments in this thread as well. The following was among the most troubling of your remarks, and the most disappointing to me personally:

...Immediately as Nvidia left our development program, getting them back into the program was raised to a very high priority. Our highest priority naturally is and remains the development of high quality and impartial benchmarks.

I'm trying to get you readers to understand what a great thing it is for us that Nvidia re-joined our development program. I think this is the best thing our company has achieved since we launched 3DMark03. The Futuremark employees went through some tough times back there, but we're all happy to get Nvidia back, including AJ (even though some weird rumours tell differently) .Immediately as Nvidia left our development program, getting them back into the program was raised to a very high priority. Our highest priority naturally is and remains the development of high quality and impartial benchmarks.

I have to tell you I find this statement simply baffling. nVidia belonged to the 3dMk03 development program for at least 15 months out of the 18-month 3dMK03 development cycle (as they didn't quit the program until December of '02.) What was it, precisely, that you could obtain from nVidia by "getting them back" into the program that would allow you to make 3dMK03 a "high quality and impartial" benchmark that you failed to obtain from nVidia during the long months in which it belonged to the FM program during the 3dMK03 development cycle? I simply cannot imagine what that might be. Please elaborate.

nVidia was in the 3dMK03 development cycle long enough to learn how to cheat the benchmark with dispatch when it shipped--it boggles the mind how you might think that getting nVidia back had anything at all to do with making 3dMK03 a "high quality and impartial" benchmark, since presumably that's what it was when it shipped.

How is nVidia being in or out of the program relevant to 3dMk03 being a high-quality benchmark as you define it, since nVidia was a full partner for virtually the entire development cycle of 3dMK03?

Hopefully, you are not declaring that 3dMK03 when it shipped was anything but a high-quality and impartial benchmark...? This is baffling for me, honestly.

I'm trying to get you readers to understand what a great thing it is for us that Nvidia re-joined our development program. I think this is the best thing our company has achieved since we launched 3DMark03. The Futuremark employees went through some tough times back there, but we're all happy to get Nvidia back, including AJ (even though some weird rumours tell differently) .

It should be obvious why Nvidia is so important for our development program, but I'll repeat some of the key reasons for those unfamiliar with our company's business.

In order to produce top quality forward looking benchmarks, we need the input and cooperation of all the major players of the industry.

So, how is it you did not receive "input and cooperation" from nVidia for 3dMK03 development when nVidia was paying you for the privilege of providing such throughout the 3dMK03 development cycle? I'm sure I don't have to remind you that the events which occurred prior to nVidia leaving your program, prior to 3dMk03 shipping, prior to nVidia cheating the benchmark, and prior to your audit report detailing exactly how they cheated it--prior to these things nVidia was just as much a part of the FM program as it is today.

Your supposition seems to be that you require nVidia's participation to be able to write a good benchmark, and yet you had nVidia's participation when you wrote 3dMK03 and before *nVidia* made the elective decision to pull out and discredit your company's software.

So if nVidia was unable to assist you in writing a high-quality, impartial benchmark prior to resigning from the program, what has changed so that now nVidia's participation is *required* to produce that result?

How could we otherwise get the highly confidential information of where the industry is going next? Each 3DMark version is aimed at the new if not next generation of hardware, and it is of crushial importance to have comprehensive information of that next generation, in order to make a benchmark for it in advance. In addition to making just high quality benchmarks, they must be impartial, and it is not convincing to release an impartial benchmark developed in cooperation with just one of the major IHVs. If we would not have got Nvidia back, we would have still done our best to optimize it also with Nvidia's upcoming products in mind, but it would have been way harder.

Again, this does nothing to explain the puzzling issue you've raised, which is why you failed to obtain that "comprehensive information of that next generation, in order to make a benchmark for it in advance" from nVidia during the development of the 3dMK03 software (since nVidia belonged to the program for the bulk of the development cycle.) Of course, the truth is you did obtain it, but that nVidia wasn't happy with what you did with the "comprehensive information" and so it quit your program. That's my take on it based on what you've said.

Of course you've glossed over the very odd structure of your business model--which is that companies pay you for the privilege of providing their own next-gen information to you so that ostensibly that information will be used in the construction of a benchmark which will portray their products in a positive light and help them sell those products. Hopefully, you can see the glaring conflict of interest in doing that and in providing a "high-quality, impartial" benchmark for the 3d-card buying public. To that end I can see nVidia's point in quitting--their view was that they were paying you for specific services which you were no longer providing them. It's the conflict of interest inherent in your business model which has caused all of this to happen, IMO.

Sadly, I must regrettably conclude that nVidia's participation is a requirement for your company because of nVidia's financial contribution to your company based on the membership fees it pays you. Obviously, I guess you've agreed to start giving nVidia its money's worth and nVidia is now back in. Seriously, unless you guys start doing some serious "rethinking" about your business model and understand the inherent conflict of interest within it I don't see much of a future for you in this endeavor, and I see more of this same sort of trouble for you on the horizon.

Well Said...

 

[WaltC]

Great posts Walt. I would have to agree with everything you said. In fact, if I were still writing benchmark articles for JPA I would have written the same thing. In fact, why don't you forward your comments to my former boss which still works for Jon at Jon Peddie Research? Her email address is kathleen@jonpeddie.com. Tell her I sent you and maybe she'll include them in next week's TechWatch Report.

Anyway, FM really needs a reality check. During the time I was doing beta testing for 3DMark since before the creation of Futuremark, there was no cost to IHVs to be part of the program that I was ever aware of. I understand that they need capital to pay for development costs, but they should seriously reconsider their business model. I remember the day that Nathan Harley came to San Francisco to meet with me, my boss and Jon Peddie about their plans for Futuremark. I was excited and genuinely liked the company's new ideas. I don't know when they started making IHVs pay to be part of the program, but I would have to say that probably started their demise. It's pretty bad when you fire your whole Toronto(marketing) office which included a founder, but to also have another founder leave to work on MP2 and yet another to leave after this fiasco. I want everybody to know that I was once approached by Nathan and AJ to work on the 3DMark program. Things didn't work out. I guess it was bad timing, but I'm kinda glad it didn't work out because I don't think I could have stayed with the company knowing what's happened recently. I'm even ashamed that my name is still listed in the 3DMark credits. In my eyes, FM has only 2 things they can do to regain my trust...

1) Quit accepting money from IHVs to be part of the 3DMark BETA program. Make your money from end-users, OEM bundles, selling of benchmark data, support services and custom demo services. If that's not enough, then think of other ways to make money, but do not continue the conflict of interest by requesting thousands of dollars from IHVs.

2) Stand-by your original stance posted in the white-paper. This means actively searching for cheats and optimizations. If any are found then they need to be rejected publicly and results removed from the database. In other words, grow a spine and don't let the IHVs push you around. The point is that all hardware should run the exact same code. If you want to provide additional tests that are not included the score, but also allow shader replacements, then that's fine, but DO NOT ALLOW application specific be used solely for increasing scores or performance.

Tommy McClain

Thank you, Tommy. I found your post fascinating--very interesting, indeed. I'll tell you, since you and I share the same opinions here, but you have some in-depth experience with some of the principals involved which I lack, it might be better for you to forward these sentiments to Kathleen as your being able to flesh out the background adds a worthwhile dimension (or, if you'd rather, feel free to reference me to whatever degree you'd like.) I, of course, agree with your conclusions as well. What puzzles me is that what's so obvious to us apparently isn't to FM. I guess we can only hope they'll do some soul searching about where their present model appears to be taking them.

Well Said...

And thank you, too, D....

 

[WaltC]

I don't see the conflict in your first point, Walt. The public could care less about the black box that is game code and hardware features and drivers--they just want a good-looking game at playable framerates. ....

Pete, what I was talking about there was the public expectation that running 3dMK03 on competing 3d hardware made by different companies will provide them comparative results which tell them something meaningful about those products in relation to each other (which they assume they can infer by looking at the numerical scores the benchmark puts out--FM's "hall of fame," etc.) But since the IHVs contribute financially to FM they each expect that the resulting benchmark software will produce scores flattering to their products, since otherwise the fees they pay to belong to the program would not be justified from their point of view. Therefore the IHVs have one expectation, the public which runs the benchmark another. Thus, the conflict of interest.

 

[BetrayerX]

@Futuremark guys (worm, Patric, and whoever else).

I paid for 3DMark03 because at the time it provided honest and meaningful results for my videocard reviews. Now, after the deals made with nVidia to allow optimizations for the benchmark, those results are difficult to understand and compare. The fact is 3DMark03 simply can't be trusted to provide those honest results anymore. Therefore, through Futuremarks own actions (and inactions), the 3Dmark03 program is now worthless and my money has been wasted.

What I'm interested in seeing is how the FM people react to such a post, because there are a whole bunch of similar minded folk out there right now who are waiting for an answer/some reassurance and it's another case of the silence being all deafening.

Well, 'cept for nVidia who keeps floating rumors... :?

Seriously, does FM plan to react to this or address it in some way soon?

If FM changed their stance about the banchmark. Ain't that a breach of the agreement you supposdely signed (when clicked yes), invalidating it? Thus, can a refund of the money be asked?

 

[BetrayerX]

Re: Ops,"I" did it again. Why does Futuremark stil

Fred da Roza wrote:
How do you fight a company that has no ethics and another that is willing to sell theirs? This is a serious dilemma for ATI. If Futuremark won’t enforce the validity of their benchmark, does ATI have any other choice but to cheat as well? If it destroys the usefulness of Futuremark's benchmark so be it. They made their bed.

Yeah, let the cheats begin.

Ladies and Gentlemen:
In the left corner we have the 800 pound gorilla...the master of all cheats, the discrediter of benchmarks, the optimizer of useles tools....Ladies and gentleman, let's give a big welcome (right) to NVidiaaaaaaaaa!

Who can beat that beast? Quackmark03?
LOL, if ATI comes with something like this (and I hope they do), I dare someone to call them cheaters or something n those lines. I'll go personally to his/her home to kick his/her ass.

 

[Mark]

@Futuremark guys (worm, Patric, and whoever else).

I paid for 3DMark03 because at the time it provided honest and meaningful results for my videocard reviews. Now, after the deals made with nVidia to allow optimizations for the benchmark, those results are difficult to understand and compare. The fact is 3DMark03 simply can't be trusted to provide those honest results anymore. Therefore, through Futuremarks own actions (and inactions), the 3Dmark03 program is now worthless and my money has been wasted.

What I'm interested in seeing is how the FM people react to such a post, because there are a whole bunch of similar minded folk out there right now who are waiting for an answer/some reassurance and it's another case of the silence being all deafening.

Well, 'cept for nVidia who keeps floating rumors... :?

Seriously, does FM plan to react to this or address it in some way soon?

If FM changed their stance about the banchmark. Ain't that a breach of the agreement you supposdely signed (when clicked yes), invalidating it? Thus, can a refund of the money be asked?

Hah, funny you should mention that. I asked for a refund from FM for exactly that reason, to which they denied. I would post the email exchange I had with them, but I'm not sure they would appreciate it... I'll think about it anyways...

 

[BetrayerX]

Who cares about what they (FM) appreciate?

Did they cared when they changed politics like a pair of dirty socks? (actually they took out the clean ones then got the dirty ones...you get the point).

I wonder if someone might sue FM for fraud.....thats what they did, basically....the change of politics is so 180 deg that no one can deny it KILLED 3dMark's credibility, and of course, usefulness.

 

[BetrayerX]

Worm[FutureMark] wrote:

NVIDIA re-joined our Development Program, which is the same as our Beta Program. The NVIDIA logo will appear on the Beta Program page asap. Refresh the page in an hour or two!

So, NVidia paid only God knows how many hundreds of thousands just to have the logo on their website?

Cut the crap, we have been swiming in it for more than 6 months now.

 

[BetrayerX]

]

I'm sorry, but none of that makes me feel any better about me spending my money on a copy of 3DMark03. Frankly, I feel like I've been cheated out of my money... no not optimized out of it, cheated. Where can I apply for a refund?

I fail to understand why you feel like you have been cheated?

Because you changed the rules of how the benchmark was to be run AFTER they purchased it, and some of us feel that by changing the rules you've invalidated 3dm2k3's usefullness as a tool...as well as all the rampant speculation about upcoming favoritism on FM's part towards nVidia.

Has ATi contacted you people yet today? I'm kind of waiting for that shoe to drop....

I wonder if ATI could potentially sue FM...they could also ask for a refund too (and a biiig one I bet)

 

[micron]

BetrayerX =

 

[AzBat]

Thank you, Tommy. I found your post fascinating--very interesting, indeed. I'll tell you, since you and I share the same opinions here, but you have some in-depth experience with some of the principals involved which I lack, it might be better for you to forward these sentiments to Kathleen as your being able to flesh out the background adds a worthwhile dimension (or, if you'd rather, feel free to reference me to whatever degree you'd like.) I, of course, agree with your conclusions as well. What puzzles me is that what's so obvious to us apparently isn't to FM. I guess we can only hope they'll do some soul searching about where their present model appears to be taking them.

Walt,

You're welcome. I went ahead and forwarded your comments to Kathleen. I gave you full credit, but if I had your email address I could have sent that along too. Anyway, I'll let you know what I hear.

As for FM, I did receive a PM from one of my contacts at FM. Evidently he wasn't too pleased with my comments, but like I told him it's a good thing I'm no longer in the business of writing benchmark articles for JPA. Personally I would had hoped that they would understand where I and others were coming from, but I believe the problem they're having is mainly staying alive through all this mess. If I was in charge I think I would rather die fighting and keep my honor and integrity instead of worrying about losing my job. Jobs come and go, but it's hard to get back honor and integrity after you've lost it. If anybody knows about this, it's me as I was "laid off" by JPA. I'm not proud of how I left the company, but it's a good thing I had long and great relationships with them or it could have been a lot worse. Anyway, hopefully some of these comments will sink in their head.

Tommy McClain

 

[WaltC]

...As for FM, I did receive a PM from one of my contacts at FM. Evidently he wasn't too pleased with my comments, but like I told him it's a good thing I'm no longer in the business of writing benchmark articles for JPA. Personally I would had hoped that they would understand where I and others were coming from, but I believe the problem they're having is mainly staying alive through all this mess. If I was in charge I think I would rather die fighting and keep my honor and integrity instead of worrying about losing my job....

I would hope they'd realize that the integrity of their software is at stake and has been put at risk. If the general perception of FM benchmarks becomes one of skepticism and mistrust, it won't take the IHVs long to figure out that they are not getting their money's worth any longer and they'll drop off anyway. The sum total is that FM's position will actually be much worse at that point than it was when nVidia quit. But that's the long view, and many companies today don't believe they can afford to take a long view about much of anything. I agree--if such interest conflicts are obvious to outsiders, why can't FM see them? Perhaps they just don't want to...

 

[BetrayerX]

BetrayerX =

micron =


What's that supposed to mean dude?

What would you do when a product doesn't do what it was promised to do?
Return it and ask for a refund. That's exactly what FM has done and I believe people and companies as clients and partners that have invested some hard earned money must have a say in the matter. This move is not only a simple Hypocritical decision from FM but also one where no one else was asked for even an opinion. If you as a costumer might feel ripped off by it, what can you expect from companies that endorse some huge amounts of money in FM.

 

[micron]

BetrayerX =

micron =


What's that supposed to mean dude?

You figure it out dude......