It was only a matter of time

[demalion]

There is indeed a case for 3dmark specific optimizations being part of the benchmark. It just makes it, in my evaluation, a far less useful benchmark. It becomes "just another game", except with more tools for data analysis (for example, sort of like UT2k3 with its rich, but different, benchmarking featureset and API feature focus).

My argument for this. I also feel Dave H's argument, that I refer to there and which can be found by scrolling up, reflects my viewpoint closely, in a way less dependent on context.

 

[WaltC]

There is indeed a case for 3dmark specific optimizations being part of the benchmark. It just makes it, in my evaluation, a far less useful benchmark. It becomes "just another game", except with more tools for data analysis (for example, sort of like UT2k3 with its rich, but different, benchmarking featureset and API feature focus).

My argument for this. I also feel Dave H's argument, that I refer to there and which can be found by scrolling up, reflects my viewpoint closely, in a way less dependent on context.

I would say that it might make 3dMk "just another game which has been heavily optimized for," instead of "just another game," because the bulk of 3d games don't receive that kind of optimization attention--probably 95% of them. Here's a good example of the way most games will run, I think:

http://www.beyond3d.com/misc/traod_dx9perf/

Considering the shader-PS2.0-DX9 component, I think it tends to underscore why FM should stick to its original plan and outlaw all optimization for the benchmark--which would make it more likely, rather than less, to indicate actual 3d-game performance.

 

[demalion]

Heh, please read the provided link, Walt.

I'm pointing out that specific optimizations do not automatically make it useless as a benchmark. Notice how suitable my UT 2k3 parallel is to your concerns, and also how related it is to the related issues of what reviewers expose to consumers in their reviews and the burden of evaluating what the actual optimizations accomplish.

 

[Bjorn]

3D Mark is not a 3d game--it doesn't even pretend to be a game.

In that case, why call the four tests that make out the final score "game tests" ?

These tests also includes physics (at least AFAIK) just as a real game would do. This doesn't make 3D Mark a game but i wouldn't say that it didn't in any way pretend not to be.

 

[StealthHawk]

I'm pointing out that specific optimizations do not automatically make it useless as a benchmark.

It depends on what kind of optimization it is. One of the optimizations NVIDIA is doing is replacing shader code with less stressful shader code. When only one IHV is doing less work, this is clearly unacceptable for a benchmark- it makes the results totally useless.

OTOH, optimizing code for register usage and making the output stay the same may not necessarily be bad, although IMO any application specific optimizations in a synthetic benchmark are iffy at best. For a benchmark like 3dmark which relies on the Orb database for users to compare their scores, optimizations weaken the benchmark's usefulness.

 

[WaltC]

In that case, why call the four tests that make out the final score "game tests" ?

These tests also includes physics (at least AFAIK) just as a real game would do. This doesn't make 3D Mark a game but i wouldn't say that it didn't in any way pretend not to be.

I don't understand your point--we both agree that 3dMK03 isn't a 3d game--it's a benchmark. It can't be played--only run. Regardless of what FM calls the specific tests--they can't be played--so where's the gaming element? Conversely, what 3d game do you know of which functions like 3dMk03?... (IMO, it would be a very boring game, indeed.)

I wasn't aware that there was any confusion as to the nature of 3dMK03...if it *is* a 3d game--I'd love to learn how to play it... As it is I only know how to run it as a benchmark.

 

[Bjorn]

I don't understand your point--we both agree that 3dMK03 isn't a 3d game--it's a benchmark. It can't be played--only run. Regardless of what FM calls the specific tests--they can't be played--so where's the gaming element? Conversely, what 3d game do you know of which functions like 3dMk03?... (IMO, it would be a very boring game, indeed.)

I wasn't aware that there was any confusion as to the nature of 3dMK03...if it *is* a 3d game--I'd love to learn how to play it... As it is I only know how to run it as a benchmark.

Yes, it's rather obvious that 3D Mark is not a game and doesn't pretend to be a game in that sense. But as i see it, it definitely tries to resemble a game afa benchmarking goes and that's why i thought the "it doesn't even try to resemble a game" was not entirely correct.

 

[demalion]

I'm pointing out that specific optimizations do not automatically make it useless as a benchmark.

It depends on what kind of optimization it is. ...

Err...exactly. Heh, I guess you're not as familiar with my viewpoints on such issues as I am with yours, which is something that I think will be addressed if you follow my initial link (I don't like re-typing my commentary again, and it should be just as easy to read there as here. ).

 

[Myrmecophagavir]

]The list is generated. It is not made manually. We only take a look at (sort of preview) it before it goes online.

Ah, you do have a sanity check then. Thank you.

Well hurrah and all, but, you know, what happened the first time round then?

 

[nyt]

Seen in FM latest news (top article on their front page):

]http://www.futuremark.com/news/?newsarticle=200308/2003082904#200308/2003082904:
So, we look only at the FX5600Ultra and 9600Pro - the view is marvelous! From that it's absolutely clear why the slightly overclocked revision to FX5600Ultra was released. The 10% performance boost allowed to win the crown back to NVIDIA at the overall scores.

What's the point of retiring the flawed HoF if it's to publish such biased snippets on the other hand? worm conveniently omitted the conclusion of the article:

"Even with the slower (and thus cheaper) memory, the RV350-based video card (600 MHz versus 800 MHz) demonstrates some superiority over the NV31 new core."

'nuf said...

 

[Doomtrooper]

 

[WaltC]

Seen in FM latest news (top article on their front page):

]http://www.futuremark.com/news/?newsarticle=200308/2003082904#200308/2003082904:
So, we look only at the FX5600Ultra and 9600Pro - the view is marvelous! From that it's absolutely clear why the slightly overclocked revision to FX5600Ultra was released. The 10% performance boost allowed to win the crown back to NVIDIA at the overall scores.

What's the point of retiring the flawed HoF if it's to publish such biased snippets on the other hand? worm conveniently omitted the conclusion of the article:

"Even with the slower (and thus cheaper) memory, the RV350-based video card (600 MHz versus 800 MHz) demonstrates some superiority over the NV31 new core."

'nuf said...

Well, when you consider what they ran on Thursday:

http://www.futuremark.com/news/?newsarticle=200308/2003082805#200308/2003082805

...it's not so bad...

 

[John Reynolds]

I wonder if ATI management is considering dropping out of FM's beta program now?

 

[WaltC]

I wonder if ATI management is considering dropping out of FM's beta program now?

Frankly, I don't understand the IHVs' attraction to it at all. IMO, I think both of them far over-estimate its importance to them in selling 3d cards. Perhaps they use it as a vehicle to guess each other's future directions and implementations...?

 

[John Reynolds]

The September issue of Maximum PC has their Dream Machine of 2003 article and what graphics card do you think they picked and based on what criteria? And do you think it really scored over 7,000 in 3DMark03 (page 29)? How many people will read that article and think: "That's the card to buy this fall!!!"

 

[martrox]

While thinking about what motivates IHV's in concert to FM, think about this: Just WTF is (and has) been going on with FM in concert to IHV's? These guys have been all over the place. First, they were nVidia's darling, back in the original GeForce days through and including the GF4's. Then, in a bid to finally appear objective, FM decides to grow some cajonies and actually start to support DX, rather than just one IHV . Then, just when FM looks the part of a real hero, they cut off the cajonies and becomes nVidia's whore again. It's pretty incredible to believe these guys will ever be taken seriously again. Maybe FM should go into politics...... they certianly have the correct moral fiber for it.

 

[Bouncing Zabaglione Bros.]

I've given up on 3Dmark - not because of the technical aspects of the benchmark, but because of the attitude of Futuremark and their inability to remain objective, or to stick with the independent ideals they profess to have. IMNSHO, FM are an unreliable, untrustworthy company, ever since they decided to throw their lot in with the unreliable, untrustworthy Nvidia.

I've had Futuremark products on my PC since day one, and for years I've had their software on my hard drive. Not any more.

 

[WaltC]

The September issue of Maximum PC has their Dream Machine of 2003 article and what graphics card do you think they picked and based on what criteria? And do you think it really scored over 7,000 in 3DMark03 (page 29)? How many people will read that article and think: "That's the card to buy this fall!!!"

But John, this assumes that paper trade mags like MPC are the only source of information for the bulk of its readership. Obviously, the rise of the Internet--indeed, its ubiquity--over the last few years has put a serious crimp in the authority and influence of the paper trade mags in the market. Just because a paper mag recommends this or that no longer equates to an automatic assumption that such recommendations will in fact be heeded by even a majority of its own subscription base. Why? Because that subscription base is also looking at the Internet even while it reads the magazine. So the question is one of whether any advice provided in a paper mag will be heeded if it is contrary to what its subscription base is reading on the Internet.

The biggest problem for paper trade mags going forward is the lead time. Whereas product news on the Internet is observable almost as it occurs, paper mags often highlight info that is as much as 60 days old when it arrives in your mailbox. This was A-OK and the status quo before the ubiquity of Internet access for most people--especially people interested in the purchase of computer peripherals. But IMO these days depending on paper trade mags for product info is very nearly passe' and almost always redundant. Paper mags have been able to offset that advantage to some degree by offering CDs full of patches and demos which are still attractive to people who do not yet have broadband. I dropped my last paper subscription a couple of years ago when I went to broadband and, indeed, the only reason I continued to subscribe to paper mags up to then was because of the CDs which were valuable to me when I was doing 56K telephony. Otherwise, I'd have dropped paper subscriptions 3-4 years ago. This isn't to say that paper mags have no influence--I'm sure they do. But I'll wager that influence has been diminished a great deal from what it was just four years ago.

Another problem with the paper mags is that besides being late with the information, the quality of the information rarely is comparable with that available on the Internet from a variety of sources. Then there's the fact that with falling subscriptions the ratio of advertising to content has been skewed such that sometimes there are more ads than articles--and also that the number of pages in paper publications seems to continuously decline. As such, I'd say there's a good chance that what people would say instead is, "I know they recommend this card, and I've read several supporting articles on the Internet about that card already, so I'll buy it," or "They recommend this card but I've read too many negative reviews on these products on the Internet, so I'll pass."

More realistically because of the time lag relative to the Internet, I can see people going ahead and making a choice and buying their 3d cards before the magazine making its recommendations even arrives in their mailboxes...

 

[Bouncing Zabaglione Bros.]

As Waltc says, the sort of computer-geek who's going to drop $400 on a graphics card is already hooked up to the net and has already done the research online. Nowadays, anyone who makes their buying decisons based on what the paper magazines say isn't the sort of person buying $400 graphics cards.

The people I know who do still read the mags would not be in the market for that kind of product, and if they were, they'd be asking *me* for recommendations on what to buy before trusting their money to a magazine review.

 

[digitalwanderer]

The people I know who do still read the mags would not be in the market for that kind of product, and if they were, they'd be asking *me* for recommendations on what to buy before trusting their money to a magazine review.

Agreed. I can't count how many times someone has asked me for a recomendation because the PR on cards can be over-whelming after a while.

I think the print media is dying. It ain't dead yet, but the end is in site.