The Way its Meant to be Reviewed?

Synthetic benchmarks show the correct results on ATI hardware which has it's built in optimiser. ATI don't need to hand tune because *all* games are improved in their shader compiler.

Which product would you buy? The one that gives you good performance on all games, or the one that gives you good performance on the top few games of the year *if* Nvidia have decided to hand optimise the code?

It's nonsensical for Nvidia to find itself in the position that it is in. It's like Intel or AMD trying to rewrite every application because their default performance is so far below par.

Are Nvidia just going to keep hacking games for years? What happens when they bring out NV40 - will they stop hand tuning games for everyone that bought a NV3x and just tell you to upgrade?

 

So good luck if the game you like to play doesn't happen to be in the "10 most popular" list.

Or doesn't have a benchmark mode built in.

 

Geeze, rev...

Don't you ever post anything other than your "industry-politics-according-to-you?" This is not quality, it's, well, opiniated politics.

 

You're right on a few points there. I was being presumptuous, but again I get to that point because that's 99% of what I've heard from readers...they want to see real games reviewed. I have no scientific poll so it's admittedly not perfect but it's all I have to go on for now. Are the readers that have PM'd, posted and or emailed me wrong (or fundamentally flawed ) then in your eyes? (Hey you even said yourself that "I've seen lots of people take the position...") I mean, I see your points, they're all very valid and I agree with them in some ways. However, I'm going primarily by what I hear directly from readers and not from what others on the outside looking in presume. So in that sense I'm not really taking a "prescriptive" attitude am I? I'll be happy to be proven wrong and gladly adjust though.

I agree that as reviewers our job is to present the reader with as much information as possible, as clearly and concisely as possible. B3D does an excellent job at this. My previous post was basically me wondering out loud about how to write reviews in a more "common ground" sense...not too basic and not too technical. I'm splitting hairs here though so enough of that.

Oh I know that my "people play games, not 3dmark" comment is as tired as "rock on!" or "who's your daddy"...I didn't intend to shake the world with that comment. I was tired so I made a tired comment. However, you said this:

Umm...I'm not so sure about that. I don't think that's the most accurate statement I've ever read let's just say. If you'd made that about AM3 then, well, maybe it would be a bit more spot on.

The rest of what you said seemed sound to me and brings up some good things to think about. Nice post all around.

 

Maybe that's why it's in the forum where people (gasp!) state their opinions... What's pertinent is that it is an opinion that significant developers agree with.

 

LOL! With 4000+ posts, your disease can spread easily, y'know?

 

Bouncing Zabaglione Bros., Joe's just giving me a return jab.

 

See this post.

Anyway, it's actually not even particularly important that most or even ANY developers agree with Rev...what's important is that Rev states the reasons for his opinions. (Which he did...) The fact that several developers tend to agree with him (and me, btw), does lend more validity to those reasons, of course.

 

Bastard! Now I gotta clean the coffee off the monitor that I spit all over it laughing!

 

The teacher that I'm working for this period is wondering what's so funny about oxidation-reduction reactions. Kudos, Joe.

 

Another opinion piece

Why is the word "optimization" still being so abused? The word is consistently being used in a very IHV-centric way, instead of a consumer and a, mostly, developer-centric way.

By this I mean that from an IHV's perspective (at least of the most selfish extreme), an "optimization" is anything that makes their product look better (whether that appearance is real or not)...which, in the limitations of what most sites are willing to investigate, and given what IHVs can accomplish if they bring significant resources to bear on obfuscation, consists exclusively of "still screenshots" and "reported fps results".

This leaves quite a barn-sized door (the Door to Fiction Replacing Fact) open for outright falsities in representing 3d interactivity and interactive image quality, as the relationship between these two metrics, as well as actual evaluation of image quality in motion, require a lot of extra steps and analysis beyond the existence of those two metrics. There are still a lot of people who simply don't understand how misrepresentative these two metrics can be (or are still actively being told they can't be, and are believing it)

Regulated and dedicated applications whose goal is to accurately measure performance characteristics actively shrink that door down in size. Their job is made easier by only dealing with facts, instead of creative fictions represented as such, which results in measurements that vary so much according to IHV interests they become meaningless.

Whether the smaller door is manageable or not, or even whether it is going to be sought out, is up to each writer and site, and the amount of effort they extend in this direction.

Games, however, only (might) do so incidentally. Their creators, AFAICS, have no inherent interest in facts in performance metrics, unless the fiction either places the blame on them or dilutes what they do have an interest in (conveying a quality game experience to a customer). This is not to say they might not have a personal preference against fiction being portrayed as fact, but that can't easily be justified as something they can dedicate themselves to regulating and dedicating significant time to.

The most significant (upcoming) occurrence of this would seem to be HL2 once it is finally available, due to its apparent: massive amount of shaders; higher standard targetted level of concurrent special effects support; and, most especially, its intended customizable and upgradeable nature (makes it somewhat similar to an open source shader benchmark framework, actually).

So, dismissing synthetics and the inconvenient games as relevant seems to make perfect sense from a certain perspective.

...

In more general detail:

The opportunities for misdirection include: fooling people into believing that only the first, larger door exists, so they accept it; making people think the larger door is smaller than it is, so that working to reduce its size seems like "much ado about nothing"; placing obstacles in the actual ability to reduce the size of the door/opportunity for fiction to be substituted, so less people actually have the time to do it.

The opportunities for addressing these include: proving clearly what has been delivered through these doors; educating people about the significance of the size of the door in a way not too much more complicated than "fps results" and "still screenshots"; catching the IHV red-handed in placing obstacles, and relating it to any of the above you might be able to do, hopefully in a timely fashion so people don't forget.

Now, how hard are these efforts in relation to each other? That is determined by how hard it is to hide, how hard it is to discover what is hidden. Also, how general human nature prefers being educated to being told they know all they need to. Not a pretty picture for trying to do the latter, IMO, but the amount of success so far has been admirable.

Neither is the picture of the proportion of resources available to the parties pursuing each, at least without a wide body of varied synthetic tests available.

Even after all that, you still haven't eliminated the door or made it small enough for all reviewers to handle, and that's the essence of why, IMO, constant new methods and improvements in measuring synthetics is not just more information, but necessary validation for other information as well.

...

What happened in this article is that we heard from game makers whose ambitions and visions for their projects, and/or (IMO, "and") personal preference for consumers being more educated, lead them, of some degree of necessity, to similar opinions. I believe nVidia was asking you to find people who don't have the same opinions...after all, there are bound to be developers whose amibtions fit more within what nVidia can deliver with developer relations, or who simply view some other aspect differently, and their comments would also be "opinions", after all.

This response to the article entirely sidesteps the issue of fact or fiction being conveyed to consumers, and makes things a matter of the opinions of "one game maker" versus those of "another game maker". I.e., it doesn't tackle the reasons the quoted developers gave for their opinions.

The entire issue of how large a door is left open to misrepresentation, and how many reviewers have the capability to deal with what is delivered through such a door to find accurate performance data, is something that isn't tackled by the article, and what would have to be tackled in order to conclusively counter nVidia (as presumed as the IHV responding) quoting other developers, as they seem likely to do in future response as word of this article widens in impact. Unless you have a very large body of direct evidence, collected against any efforts and resources dedicated to prevent your finding it, such a discussion would be largely political, which is something Beyond3D tries to steer clear of.

Don't be surprised if your dedication to avoiding such a political discussion is used against this article to portray why it should be dismissed as unfair and not truly representative of differing viewpoints...aka "biased". Willingness to enter such a political discussion would be attacked as well, as long as clear proof could not be offered.
The problem here is you are biased against nVidia's interests, because your interest is in actual hardware performance characteristics, and nVidia is comfortably established in aggressively treating the first as more real than the second. This has demonstrably not been reconciled, and will not be until nVidia's attitude on the matter changes, the reality of their hardware changes, or your comments are aligned with nVidia's interests or no longer address areas counter to them, ahead of either of the first two changes coming to pass.

IOW: What's the next pdf going to look like?

 

Thanks for explaining the in-joke Of course it does probably indicate that you need to spend less time online....

 

Tim Sweeney (Epic Games) wrote:

"Developers aren't going to provide you with benchmarks at least until they're reasonable close to shipping, and that's a good thing, since benchmarking hardware with very much unfinished and unoptimized software is going to produce unreliable results."

Danny Lepage (Ubisoft) wrote:

"Practically, what this means is that we rush features in and we barely think about optimization for a while. Would that be worthwhile to benchmark video cards (or CPUs) with this kind of engine? In my opinion, the answer is no."

Then, you read the article and see the spin that B3D portrays on the "beneficial" synthetic benchmarks. What a load of sh**! Clearly it has been pointed out by Tim and Danny that you'd have better luck testing with Alpha's instead of synthetic benchmarks. The most simplistic answer would be is what was said already by Tim, "benchmarking hardware with very much unfinished and unoptimized software is going to produce unreliable results."

When you take out the driver developer and game developer from the equation, you get a false positive or negative. It is these two teams that make adjustments to the performance that dictates how well a videocard can perform. You can have the most cutting edge hardware out there, however if you don't have software written to utilize the hardware... all that effort on the hardware means squat!

 

Y'all are missing the point. If the software is coding to the DX standard it should run a DX synthetic benchmark just fine, if it needs to be specially coded to run a standard DX synthetic benchmark then the problem lies with the need to be specially coded and not with the benchmark.

 

I think you have this backwards.

 

The problem is you are not going to change how this process works. It is a method of developement that has been used to be the most efficient that it can be in a given time. All software needs to be patched or updated. Simple saying "If they just followed spec, everything would be peachy" is completely ignorant. It is the software that drives the hardware, not the reverse.

Nothing you say or attempt to portray is going to to work out. The reason why it isn't is due to the fact that game developement does not all happen at the same time. Features are added as they become availible so that the game will have a longer life cycle. That is why all games are not coded to DX9 spec and why it won't happen. That is also the reason why synthetic benchmarks are that much more invalid. "It is wonderful that Card A obliterates Card B in this one features of DX9. Too bad that one feature is rarely used." See how that works? Because a piece of hardware is designed better for DX9 spec doesn't mean it is going to have the support in games to prove it is the better card. Software needs to be Optimized.

 

Where did you see anything in what they said that indicates that synthetic benchmarks are "unfinished and unoptimized" software?

It seems to me that the only way you could take Tim and Danny's statements above, and read into them some implicit condemnation of all synthetic benchmarks as "unfinished and unoptimized" is if you had already made up your mind that this is the result you want, and are desperately trying to twist any information to back up that point of view.

Tim and Danny were talking about the development of their software and why they would not consider distributing early Betas to use for benchmarking. This says absolutely nothing about the validity of the results from a finalised version of a synthetic benchmark.

In fact, the comments that you quote above say nothing about synthetic benchmarks at all.

Let's look at some other quotes -

Sounds like a fairly firm endorsement of the use of synthetic benchmarks to me.

So Danny seems also to be endorsing the use of synthetic benchmarks, although he clearly feels that there are improvements to be made. He also makes what I believe to be a very good point, which is that it would be a good thing if game developers could make themselves more involved with the creation of synthetic benchmarks - after all, they are the ones who know what they plan to do with future engines, so their experience and advice would be an invaluable input to the development process of these benchmarks.

This is where sites like B3D could perhaps be of service - by filtering information from developers to companies like Futuremark to help them to stay firmly in touch with the directions being taken by the game industry.

 

pssst. no one is saying that being faster in a single feature equals the better game experience. people are saying that you can at least make a conjecture about future performance based upon the performance of individual performance. if a game makes heavy use of the DX9 feature where Card A obliterates Card B, it will run faster with Card A, no matter what the optimizations on Card B are. that's all people are saying. synthetics can provide information beyond the barrage of game benchmarks, and an intelligent individual can use that information to form an opinion about performance in future applications.

 

This clearly means that Danny doesn't think that using one of his game engines for benchmarking hardware is a good idea. Because of the "rushed" nature of high end features, there is no way that his games can stress the new features in graphics cards.

Here he is obviously advocating the use of synthetic benchmarks to stress and measure graphics hardware because games (made with the aim of being games and hitting release dates) do not perform the same function as software designed to measure and stress the capabilities of the graphics hardware.

Using synthetic benchmarks is analageous to a car magazine taking a car to a race track. They measure top speed with radar traps, they measure cornering at 150 mph, they measure 0-60 times and so on. These are all useless to the average driver. You cannot drive like this around town.

However, these measurements are still a useful metric that tells us different information, that is still useful and interesting to consumers, even if most people (but not everyone) will only see a fraction of what is shown on the odd occasion when they try to leave the traffic lights quickly.

Poor results in synthetics are harder to fool and explain away, hence Nvidia completely disavowing them alltogether. Nvidia have been able to hide poor results in the few popular games used as benchmarks by a hurculean and short term cheating campaign that is much easier to hide.

Once again, Anonymous has shown us that most people who rubbish synthetics are doing so based on the misinformation that originated from Nvidia and their pet websites. You do not understand the context and meaning of synthetic results, or what extra information they can give us. You parrot Nvidia's 3DMark-trashing PDF fourth-hand, without understanding what information you are losing - which is exactly what Nvidia want. Why do they want this? Because synthetics in particular show how weak Nvidia's current cards are compared to their competitiors.

Even games that have come out using advanced features that show poor performance on Nvidia hardware have been brushed off as "poorly programmed" in the same way, and with the same lame excuses as Nvidia have used in their arguments against synthetics. Why do you suppose that is? Synthetics are "poorly programmed", games are "poorly programmed", etc. Is there *anything* that Nvidia endorses that actually shows the impressive performance and "cinematic computing" that Nvidia constantly claim, but never manage to demonstrate?