Synthetic Benchmarks - my view on them

[Infinisearch]

I have never really cared for synthetic benchmarks, when reading a video card review or roundup its usually the page i skip. The only thing I ever found them useful for was for seeing how close they came to their theoretical/advertised specs. With the release of 3dmark05 (which i can't run -radeon32mb- and can't download -56k-) alot of comments on the pro and cons of non-game benchmarks have been brought up about both pure synthetic benchmarks and 'demo' benchmarks. I don't think anyonelse has brought up this point of view so I just wanted to offer it up for comment.

My main beef with synthetic benchmarks is not with the benchmarks themselves but with ATI, NV,... and to a certain point at least in the case of 3dmark (i don't know of any other business models so please feel free to enlighten me) their business model's effect on the IHV's. What I'm saying is as a consumer I would rather see a company spend their money, developer relations manpower/time, and driver team manpower/time on actually games and applications rather than something that is purely made to be a benchmark. Its not a definate but, If Syn benchmarks did not exist, or media outlets decided to rely more on actual application benchmarks, the resultant shift of resources from optimizing for a specific benchmark would be free to make the consumer experience with their video cards better. By better it could mean a host of things from more stability or speed in a certain game, or maybe the same speed or stability except 4 monthes earlier because the driver team didn't have to worry about Syn benchmark 'x'. Now I know this will sound suspicious since I already mentioned I owned a Radeon32ddr (maybe I should mention i'm still running 98se) but I'll say it anyway... "better support for older cards", i don't think its to much to ask for. (Well maybe not my card but still)
I understand that benchmarks sell cards, but I think a concentration on games, applications, and customer security will not only sell more cards but garner more return customers as well.

Anyhow I just felt like writing something and this topic came to mind, and it seemed somewhat relevent with the recent release of 3dm05. Comments... agree / disagree / both?

 

[Reverend]

Do you like B3D video card reviews? Whether it's a Yes or a No -- Why?

Are you interested in 3D or are you just interested in which video card is the best for playing games?

What is the primary reason you're participating in the B3D forums?

 

[Pete]

If the benchmark is based around DirectX, not IHV-specific, features, one would expect some tweaking to be beneficial to DX games in general.

I guess.

 

[Moloch]

Do you like B3D video card reviews? Whether it's a Yes or a No -- Why?

Are you interested in 3D or are you just interested in which video card is the best for playing games?

What is the primary reason you're participating in the B3D forums?

 

[Tim Murray]

I think Rev's point is--why the hell does this thread exist? It's nonsensical, silly, and just outright ridiculous. "IHVs would spend more time optimizing for games if synthetic benchmarks did not exist?" Pass the bong. If a fillrate test shows significant gaps between theoretical and actual fillrate in certain circumstances, for example, and these oddities are replicated by game benchmarks in fillrate-intensive situations, this tells us one of two things:

1. Hardware is slow in case X for whatever reason.
2. Driver issue (not even necessarily a bug, maybe just some redundant code somewhere or something like that) causes slowdowns in case X.

Since we often have tech specs, we can tell if a card should be slow in case X or not. See why synthetic benchmarks (and why "cheating" iin them--cheating in fillrate benchmarks? what the fuck is this guy on?) is really pointless?

 

[Rys]

Do you like B3D video card reviews? Whether it's a Yes or a No -- Why?

Are you interested in 3D or are you just interested in which video card is the best for playing games?

What is the primary reason you're participating in the B3D forums?

Rev is asking him why he's here, when it's clear that B3D's GPU reviews are mostly about theoretical performance measured by somewhat synthetic tools, with analysis focussed around that. You get AIB partner board reviews here too, full of game tests and less of the synthetics, but they're not the focus.

So why complain about synthetics being used to measure card performance on a site where that's quite a large focus, something that's not hidden and not likely to change.

At least that's how I read it.

On topic, I think that IHVs have a job to look at synthetics in terms of optimisation, since synthetics (and this includes 3DMark05) are often the first port of visible call on new graphics rendering techniques. They're usually the first place to find how certain algorithms are implemented, or where new major revisions of APIs are first used and exploited.

So having a look at how those tests are using those APIs, to see if they're brain-dead in their usage or spot on (and if they are, to see if there's grounds for a generic opt), which may benefit consumers of their cards, is something an IHV has to do.

Whether certain synthetic tests have too much influence over video card sales really doesn't matter. People that just look at a 3DMark score for a card choice will generally buy the card with the best box art, after they've selected a basic product from an online review. For those people, 3DMark isn't a bad tool for them to rely on. It ranks hardware pretty well, at various price points.

Off topic a little, for the rest of the community looking to buy a video card based on informed research, there's reviews for everyone at a bunch of websites, that go into good depth, using synthetics or not. If you read a HardOCP review along with a B3D review, on the same product, I'd rarely say you'd come away with conflicting opinions on that product from both sites, plus you've just assimilated (hopefully) broad, good knowledge on the hardware you want to buy.

Synthetics and games both have a role to play in evaluation of consumer 3D hardware. Thinking otherwise is folly.

Whether you like synthetics or not is besides the point. They're still highly useful and will never disappear. You'll just end up giving sites like B3D a miss (which is your loss) if you hate them that much.

Rys

 

[ET]

What I'm saying is as a consumer I would rather see a company spend their money, developer relations manpower/time, and driver team manpower/time on actually games and applications rather than something that is purely made to be a benchmark.

Let me chime in. We're all against companies cheating on benchmarks, if that's what you mean. But general optimisations help games, too. That's true in particular for a benchmark like 3DMark, which is intended to be somewhat representative of games. If a benchmark shows a bug in a driver, this bug is not less significant than a bug in a game -- simply because games are likely to fall on this bug, too. Anyway, I'm sure that ATI and NVIDIA dedicate a lot more time to games, simply because there are a lot more games than benchmarks.

 

[jb]

Rys,

that was a good post and thanks for the thoughts!

 

[digitalwanderer]

Synthetic tests are a tool, if used properly they're useful as all get out....when used for measuring e-penises they just seem to cause trouble.

 

[Infinisearch]

Do you like B3D video card reviews? Whether it's a Yes or a No -- Why?

Are you interested in 3D or are you just interested in which video card is the best for playing games?

What is the primary reason you're participating in the B3D forums?

Well before I reply to your introspective questions Reverend let me first thank Rys for his insight to the purpose of them. It might have taken me a while as to figure out their purpose on there own because if Rys was right, my intent had nothing to do with this or any other website.

1.Yes I like B3D reviews, why... its because there more technical commentary they provide. Let me expand on this in respect to my original post by first quoting it.

The only thing I ever found them useful for was for seeing how close they came to their theoretical/advertised specs

There is no doubt (and I should add I didn't dispute) as to synthetic benchmarks having there place; and B3D's use of them is a great example of that, but in all fairness how many websites use synthetic benchmarks like that? In the paragraph following some charts of synthetic benchmarks in a B3D review you might see something along the lines of "and here we see the advantage of having a crossbar memory controller using the available bandwidth more effectively..." where as in most other websites if you actually do get a commentary after synthetic benchmarks you'll most likely see something along the lines of "and here we see IHV A with a 12% lead over IHV B".

2. Both.

3. To involve myself in discussions of a more technical nature, so that I may learn and help when possible.

Now I'm curious, was Rys's take on the reasons for your post correct and if so why did you take the post personal (or should I say as if I targeted B3D) as I just put an opinion out there for commentary?... Nothing to do directly with this or any other website... or were you just trying to see if i'd answer the questions?

Anyhow given the quote I already used in addition to the following one from my OP, I'm really wondering what the big deal is...

My main beef with synthetic benchmarks is not with the benchmarks themselves but with...

 

[Infinisearch]

What I'm saying is as a consumer I would rather see a company spend their money, developer relations manpower/time, and driver team manpower/time on actually games and applications rather than something that is purely made to be a benchmark.

Let me chime in. We're all against companies cheating on benchmarks, if that's what you mean. But general optimisations help games, too. That's true in particular for a benchmark like 3DMark, which is intended to be somewhat representative of games. If a benchmark shows a bug in a driver, this bug is not less significant than a bug in a game -- simply because games are likely to fall on this bug, too. Anyway, I'm sure that ATI and NVIDIA dedicate a lot more time to games, simply because there are a lot more games than benchmarks.

To tell you the truth I wasn't even thinking about cheating on benchmarks, and while I mentioned speed it wasn't my main concern. Stability however was and let me clarify by stability I meant it as a blanket term covering both 'not crashing', and 'correct rendering'. In fact the trigger for me writing the OP was the fact that both IHV's rushed out a driver for the release of 3dmark.

I had for a while now been looking to buy a new video card so I decided it would be wise if I checked out some forums for a the IHV of the video card I was interested in. Since I wanted a DX9 card my budget basically put me at a Radeon 9550 - 9600xt, I decided to go to the Rage3d boards and see the problems people were encountering. The short of it was I saw some problems that had persisted for months and still hadn't had resolution... Now if I was one of those people and then I saw ATI rushing out new drivers for the release of a pure benchmark, I'd be pissed. Call it empathy or whatever you will but it seems wrong to me at least from a consumer perspective.

As to your last statement I'm not to sure about that, quite simply because benchmarks help sell videocards and help consumers choose between IHV's... in otherwords its 'Big Business'. Since benchmarking is used as a form of 'indirect technical advertising' and through observed situations like the one described in the previous paragraph leads me to believe that benchmarks play a prominant role in resource allocation, but thats just me from the outside of the industry trying to look in through a frosted glass window.

 

[Reverend]

I didn't mean anything nasty with my post, just wondering if you know why you may like B3D reviews that has synthetic tests while you complain about synthetic tests, that's all.

 

[Infinisearch]

On topic, I think that IHVs have a job to look at synthetics in terms of optimisation, since synthetics (and this includes 3DMark05) are often the first port of visible call on new graphics rendering techniques. They're usually the first place to find how certain algorithms are implemented, or where new major revisions of APIs are first used and exploited.

OT - Once again thanks for the help with Reverends comment, since I wasn't thinking along them lines It would have taken me a while to come to understand there purpose. Back On Topic somehow I was always under the impression that the IHV's used their own profiling and statistical tools in optimization. As to the techniques and implementation both IHV's have developer relations people that are familiar with new techniques and implementations and IIRC do their own research as well. In addition don't both IHV's also have demo teams? Don't they keep up to date with new rendering techniques also? to showcase new hardware?

So having a look at how those tests are using those APIs, to see if they're brain-dead in their usage or spot on (and if they are, to see if there's grounds for a generic opt), which may benefit consumers of their cards, is something an IHV has to do.

Again aren't the IHV's involved with this up front...? don't the IHV's work with microsoft on DirectX? Aren't some IHV's part of the 'OpenGl' board or something like that?

Whether you like synthetics or not is besides the point. They're still highly useful and will never disappear. You'll just end up giving sites like B3D a miss (which is your loss) if you hate them that much.

Never said I hated them, never made the comments in reference to this or any other Web-Site nor did I deny there usefulness, though I do disagree with you partially in the context of where they are useful. I never said I wanted them to disappear, I only said that I wish they (the IHV's) would dedicate more of there resources towards dealing with consumer problems (aka driver issues) with real applications and hopefully with older cards rather than dedicating said resources to a program that is purely used as a benchmark.

 

[Infinisearch]

I think Rev's point is--why the hell does this thread exist? It's nonsensical, silly, and just outright ridiculous. "IHVs would spend more time optimizing for games if synthetic benchmarks did not exist?" Pass the bong. If a fillrate test shows significant gaps between theoretical and actual fillrate in certain circumstances, for example, and these oddities are replicated by game benchmarks in fillrate-intensive situations, this tells us one of two things:

1. Hardware is slow in case X for whatever reason.
2. Driver issue (not even necessarily a bug, maybe just some redundant code somewhere or something like that) causes slowdowns in case X.

Since we often have tech specs, we can tell if a card should be slow in case X or not. See why synthetic benchmarks (and why "cheating" iin them--cheating in fillrate benchmarks? what the fuck is this guy on?) is really pointless?

While I find it really amusing being painted as a Weed Smoking, Tie dye shirt wearing, bell bottom having, crazed hippie who dances around a fire on a beach while howling at the moon and sticking pins in voodoo dolls of video benchmark company CEO's...

I'm just wondering if you actually read my OP or just glanced through it? If your response was based on actually reading my OP where exactly do you get your drugs from, they sure are a whole lot stronger than any I've ever done.

 

[Neeyik]

Anyhow I just felt like writing something and this topic came to mind, and it seemed somewhat relevent...

One should expect any form of criticism when one's defense for making a post runs along the lines of "Didn't have anything to contribute, so just came up with potentially flammatory topic".

 

[Rolf N]

I don't think 3DMark passes as a synthetic benchmark. It has some (semi-)synthetic subtests, but the overall emphasis, as can be verified easily by looking at the score calculation formula, is "techdemo" benchmarking.

It's just a graphics demo, folks, like The Fly or Farbrausch-xy. And that's surely entertainment software (and, to an extent, showing off). I did use The Fly once for a "competitive review", and 3DMark is really very similar except for
a)it has much more advanced benchmarking functionality - Entech sys info and the ORB vs nothing, to be exact.
b)it has far more exposure.

A is great. B can be a problem because of these ugly "concentration of effort" phenomena happening in the driver departments. Benchmarking techdemos is usually real nice because there are so many of them. Very diverse rendering techniques, just take your pick. IHVs won't manage to optimize for them all. And because some users actually use their cards to watch tech demos (for entertainment), even if that's a minority, it's "real-world" in a sense.

As I straid a bit from my point, I'll repeat it: IMO 3DMark is not really a synthetic benchmark. MDolenc's Fillrate Tester is. You don't think these are comparable, or do you?

 

[Nappe1]

though this thread is proven to be "unnessessary", I'll add my reasons to read and use benchmarks like 3DMark series.

Usually, the GFX card benchmarks that use synthetic benchmarks as well as real game benchmarks, use only games that are extremely popular. (to get biggest audience with considerable amount of work, of course.) Too bad that, it's extremely rare that games I play, are included. 3DMark comes in handy, when you have a sort of knowledge, what performance numbers you get with your system on 3DMark and how that compares to games your play. Of course, you have to use correct 3DMark and correct settings able compare the values. for example older lots of CPU power (No HW T&L support or very limited one.) wanting games behave pretty much same way as 3DMark 2000 run in SW T&L. newer games with DX7 support match pretty well with some test on 3DMark 2001SE and so on.

The Point is, that you have know how to compare the stats. Any benchmark (not even game based ones,) cannot provide complete single value, that would immediately tell you how your computer will work on every game on this darn small miserable planet.

another thing is optimizations... Every benchmarking method that gets as popular as 3DMark is now, Will eventually face the same optimization problems as FM has been dealing with.






and at the end I'll Lighten the atmosphere a bit:
Do you know who is that guy in black leather jacket in Lobby scene on 3DMark 2001SE? (this should not be too hard, when I am asking this...)