[GraphixViolence]

Scali, you keep saying that DST shadows should be the default in 3DMark05 because this is what most future games will do. While the point is certainly debatable, that's not the real issue.

As Reverend has stated, 3DMark05 is NOT a game. Trying to make it behave exactly like a game would be an exercise in futility. But that doesn't mean it can't provide a good indication of RELATIVE game performance on different hardware. In order to do this, Futuremark must do what it can to ensure that all systems tested are displaying the same image. Games do not have this requirement - developers can make them look however they want on different hardware. But that's because games aren't designed to be used for comparison.

So how does Futuremark accomplish this, while still keeping their benchmark as game-like as possible? First, they use an industry standard API, DirectX 9. This API has strictly defined behavior. WHQL tests ensure that a given product with a given driver produces the correctt output within predetermined tolerances. Second, they approve drivers to ensure that they have passed WHQL testing and/or are free of optimizations that alter behavior based on the knowledge that a benchmark, rather than a game, is being run. Third, in the case of things like MSAA quality that are not covered by WHQL, these are not enabled by default (even though they are widely enabled by end-users) because there can be no guarantees that quality will be comparable across different platforms.

It should be apparent that Futuremark has taken many steps to make 3DMark capable of producing comparable results between different products. Now along comes DST. Enabling it produces a different image on hardware that supports it vs. hardware that doesn't. It isn't part of DX9, so it isn't tested by WHQL, so there is no industry standard that specifies how it should look. So now the image quality of all cards are bound by the DX9 specification except Nvidia, and results are no longer comparable.

Why is it so important that results be comparable? Because the whole point of a benchmark is to allow you to measure or judge the merits of something relative to some reference point. If that reference point changes depending on what product you're using then that limits the usefulness of the benchmark. In the case of DST, there is no reference that can be used to define how it should look, and so scores with DST cannot be fairly compared even to each other, let alone to other cards that don't support it. On the other hand, all cards that don't support DST, or don't have it enabled, are held to a common standard and therefore can be fairly compared.

So by supporting DST, for whatever reason, Futuremark is undermining the value of 3DMark05 as a tool to determine relative performance.

[Bjorn]

Originally Posted by GraphixViolence

Games do not have this requirement - developers can make them look however they want on different hardware.

Yes, and that's of course the problem. If most upcoming games will use this feature in the same way that FM are doing in 3D Mark 05 (which we pehaps need to investigate further) then not using it would make 3D Mark worse as a benchmark for "future games". Though better as a syntethic benchmark. But that's not what FM is about imo.

[Razor1]

Originally Posted by ChrisRay

Razor, Perhaps you could enlighten us to what exactly your point is. Or at least clarify.

Sorry went to bed lol,

Ok from what I've seen around other forums aswell, there has been an increase on the pci-e x800xt aswell with the hot fixes, around 200-300 points. I think there was a memory bug, and that was fixed which gave a substatial boost for the agp, but there are other opts done (could be just tweaks), to give the a few more points. Not saying ATi cheating, but I think 3dmark05 just went down the toilet

[[maven]]

Originally Posted by DaveBaumann

Both solutions are sampling 4 values from the depth map, however NVIDIA's solution (as its done in the texture sampler) can do the 4 point samples, the comparison and then a bilinear filter on the result in a single cycle; the alternative solution has to read the point samples in and then compare those against the depth in the pixel shader, and then provide an average of the results - it can't do this in a single cycle (although with scalat ops, it might not need to be 4 cycles for the comparisons).

The difference in output occurs becuase the process is slightly different - NVIDIA's resultant value comes from a bilinear filter on the fractal results of the comparison, whilst the alternative is an an average.

I'm not sure on this, but wouldn't both approaches give very different results on shadow map "minification". NVIDIA always takes neighbouring samples and point sampling can "skip" in-between texels...?

[MuFu]

Originally Posted by Razor1

Sorry went to bed lol,

Ok from what I've seen around other forums aswell, there has been an increase on the pci-e x800xt aswell with the hot fixes, around 200-300 points. I think there was a memory bug, and that was fixed which gave a substatial boost for the agp, but there are other opts done (could be just tweaks), to give the a few more points. Not saying ATi cheating, but I think 3dmark05 just went down the toilet

Well since 3DM05 is a step ahead of current game engines and aims to emulate future ones, it's reasonable to assume that a convenient time for IHVs to implement "general" driver optimisations for future games is now. I wouldn't be suprised if scores continue to go up over the next few releases, the result of optimisations that probably won't benefit games for months or even years.

[Razor1]

Originally Posted by MuFu

Well since 3DM05 is a step ahead of current game engines and aims to emulate future ones, it's reasonable to assume that the time to implement "general" driver optimisations for future games is now. I wouldn't be suprised if scores continue to go up over the next few releases, the result of optimisations that probably won't benefit games for months or even years.

That is very true, but, at this point, we will have to wait and see, what happens, but this memory bug, if it truely was a memory bug, would expect other benchmarks, like Codecult code creatures, or 3dmark03's nature scenes, or 3dmark05 other levels and not just the first level to get teh same boost, even Doom 3 should get a similiar boost. All of these tests use huge amount of vram due to high geomtric counts and shader counts.

[Dave Baumann]

All of these tests use huge amount of vram due to high geomtric counts and shader counts.

No, they don't. 3DMark03, for instance, was designed to operate within the memory of 256MB boards, which is probably not the case with 3DMark05 with 512MB boards due on the horizon (I've not actually read the mem specs for o5 though). Something like Doom3 is very texture intensive, and not vertex intensive - plus, OpenGL handle vertex pools differently, and much of the geometry processing is done on the CPU with Doom3 so vertex pools are not likely to be required as much.

Read my earlier post on the differences between the PCIe and AGP performances.

[Razor1]

Originally Posted by DaveBaumann

No, they don't. 3DMark03, for instance, was designed to operate within the memory of 256MB boards, which is probably not the case with 3DMark05 with 512MB boards due on the horizon (I've not actually read the mem specs for o5 though). Something like Doom3 is very texture intensive, and not vertex intensive - plus, OpenGL handle vertex pools differently, and much of the geometry processing is done on the CPU with Doom3 so vertex pools are not likely to be required as much.

Read my earlier post on the differences between the PCIe and AGP performances.

Well then have Far Cry either way both Doom 3 and Far Cry stress the Vram it thier limits, if the memory allocation bug limited the x800's 256 mb cards it would have showed up earlier.

[GraphixViolence]

Originally Posted by Bjorn

Yes, and that's of course the problem. If most upcoming games will use this feature in the same way that FM are doing in 3D Mark 05 (which we pehaps need to investigate further) then not using it would make 3D Mark worse as a benchmark for "future games". Though better as a syntethic benchmark. But that's not what FM is about imo.

This is really a lot simpler than you seem to think it is. Whatever type of benchmark you think 3DMark is, there is no doubt that it is a benchmark. And the whole point of a benchmark is to measure or judge something against some reference or standard. That is true whether your benchmark is a game or synthetic, future-oriented or not. And you can't measure something against something else without some consistent point of reference.

If I have two products, one with a 3DMark score of 2000, and one with 4000, what does that mean? Does it mean the latter is significantly better than the former at something? I'm sure that's what Futuremark would like it to mean. But if each score is measuring something different, than you can't even make a basic determination about which product is better, and the numbers become useless.

[Scali]

Originally Posted by DaveBaumann

This is what flumoxed me a bit - Scali's use of "one sample" isn't how I would quite say things.

The way I define a sample is 'whatever is returned by a single texture sampling instruction'. I believe this is common convention among developers.
I believe it is also common to speak of 'taps' rather than 'samples' when they are the (point-sampled) inputs of a filter (you might be familiar with this from anisotropic filtering).

In both cases, you effectively take 4 taps or point-samples from the depth map. The catch is that point-samples are no faster than bilinear samples on 3d hardware, so it's not all that efficient to take 4 separate samples.

[Razor1]

Originally Posted by GraphixViolence

This is really a lot simpler than you seem to think it is. Whatever type of benchmark you think 3DMark is, there is no doubt that it is a benchmark. And the whole point of a benchmark is to measure or judge something against some reference or standard. That is true whether your benchmark is a game or synthetic, future-oriented or not. And you can't measure something against something else without some consistent point of reference.

If I have two products, one with a 3DMark score of 2000, and one with 4000, what does that mean? Does it mean the latter is significantly better than the former at something? I'm sure that's what Futuremark would like it to mean. But if each score is measuring something different, than you can't even make a basic determination about which product is better, and the numbers become useless.

True, for us its not an issue, but for alot of others not so techo savy, its kinda of buying guide

[Dave Baumann]

Originally Posted by Razor1

Well then have Far Cry either way both Doom 3 and Far Cry stress the Vram it thier limits, if the memory allocation bug limited the x800's 256 mb cards it would have showed up earlier.

I'm not aware of anything that says Far Cry is memory bound - can you point me to something please?

Anyway, that doesn't mean is has the same memory requirements, nor does it mean that its will trigger similar issues, or issues to the same extent. And in the case of Doom3, as has been mentioned before, this is more texture limited, is OpenGL (and not necessarily privvy to either the same issues or the same fix yet, if its applicable) and much of the vertex processing is done via the CPU so vertex buffers are not as likely to be an issue for the graphics board.

[Razor1]

Originally Posted by DaveBaumann

I'm not aware of anything that says Far Cry is memory bound - can you point me to something please?

Anyway, that doesn't mean is has the same memory requirements, nor does it mean that its will trigger similar issues, or issues to the same extent. And in the case of Doom3, as has been mentioned before, this is more texture limited, is OpenGL (and not necessarily privvy to either the same issues or the same fix yet, if its applicable) and much of the vertex processing is done via the CPU so vertex buffers are not as likely to be an issue for the graphics board.

Vertex processing in Ogl is only done on the CPU if VBO's are not implimented. Cards previous to last 2 line that would be the case.

[hovz]

lol dave, are you annoyed yet?

[Dave Baumann]

Originally Posted by Razor1

Vertex processing in Ogl is only done on the CPU if VBO's are not implimented. Cards previous to last 2 line that would be the case.

You can code your own CPU processing for vertex data - this is what Carmack has implemented because of the shadows and the number of passes required per light if skinning was done on the Vertex shader.

[Scali]

Originally Posted by GraphixViolence

This is really a lot simpler than you seem to think it is. Whatever type of benchmark you think 3DMark is, there is no doubt that it is a benchmark. And the whole point of a benchmark is to measure or judge something against some reference or standard. That is true whether your benchmark is a game or synthetic, future-oriented or not. And you can't measure something against something else without some consistent point of reference.

If I have two products, one with a 3DMark score of 2000, and one with 4000, what does that mean? Does it mean the latter is significantly better than the former at something? I'm sure that's what Futuremark would like it to mean. But if each score is measuring something different, than you can't even make a basic determination about which product is better, and the numbers become useless.

3DMark (and most other graphics benchmarks) has never been completely apples-to-apples. For example, we have hardware that does perspective gouraud shading, while there is hardware that does linear gouraud shading. Oh dear! Now all our 3Dmark99/2000 scores are suddenly invalid aswell! Not all cards performed the exact same operations!?
Same goes for hardware T&L. Obviously cards with hw T&L had an advantage in 3DMark2000 and beyond. I never heard anyone complain that all hardware should be using CPU T&L to get a comparison.
Or what about 3DMark03 where the Radeon 8500 could render GT2 and GT3 in a single pass, using higher precision than the ps1.1-based two-pass version for GF3/GF4? Never heard anyone complain about that either. Or what about the fact that ATi cards only use 24 bit precision with ps2.0 shaders, while GeForces use 32 bit? I suppose that invalidates any possible comparison between the two. And then there are different approaches to mipmapping, anisotropic filtering and all that. Nobody ever blamed a benchmark for it. Most people probably weren't even aware of any differences until they had it pointed out to them by someone knowledgeable. I wonder how many people would even have noticed there were different shadowing methods used if it wasn't pointed out to them.

People talk about consistency. I think the only thing that's not consistent here is the way 3DMark used to be treated, and the way it is treated now.
Different hardware has different features, and renders things slightly differently. I believe the correct term is 'DUH!'.

[trinibwoy]

Can't wait for the argument to refute that last post - that was a whopper.

[Dave Baumann]

In each of those cases the differences were within the guidelines and control of the API is was being coded on - in this instance thats not the case since there are no guidelines / rules / controls within the API. The other factor here is that there are no issues with implementing something in DX since a feature that is licensed or implemented by microsoft for DirectX is available for any vendor to implement in their own hardware by way of their DirectX hardware license with Microsoft - this is not the case here and potential IP issues are already creeping out of the woodwork.

As for implementing features that are single vendor, but still within DX - previously Futuremark have always said that these are implemented when they have commitment from at least two vendors that they either have, or will have, a hardware implementation within a reasonable timeframe.

[CMAN]

Could it be possible Futuremark thought ATI would have DST in their newer cards past the current generation?

[digitalwanderer]

Originally Posted by DaveBaumann

As for implementing features that are single vendor, but still within DX - previously Futuremark have always said that these are implemented when they have commitment from at least two vendors that they either have, or will have, a hardware implementation within a reasonable timeframe.

Is XGI going to support DST? (Heck, does XGI still exist? )

[Dave Baumann]

In fact, I suspect that others will support DST - I've already heard of S3 supporting it. However, AFAIK thats only for the storage of the depth values; the fast PCF solution is another question entirely.

[Scali]

Originally Posted by DaveBaumann

In each of those cases the differences were within the guidelines and control of the API is was being coded on - in this instance thats not the case since there are no guidelines / rules / controls within the API. The other factor here is that there are no issues with implementing something in DX since a feature that is licensed or implemented by microsoft for DirectX is available for any vendor to implement in their own hardware by way of their DirectX hardware license with Microsoft - this is not the case here and potential IP issues are already creeping out of the woodwork.

As I said before, if game developers don't care about this, then why should Futuremark? Or even worse: why should you?
That point is completely irrelevant in a world that's filled with games and other software using vendor-specific extensions.

As for implementing features that are single vendor, but still within DX - previously Futuremark have always said that these are implemented when they have commitment from at least two vendors that they either have, or will have, a hardware implementation within a reasonable timeframe.

What makes you so sure that FM doesn't have that commitment from a second vendor?
And even if they don't, who are you to tell them to stick to guidelines that they made years ago, which simply don't apply anymore?

[digitalwanderer]

Originally Posted by Scali

As I said before, if game developers don't care about this, then why should Futuremark? Or even worse: why should you?

Because good or bad, right or wrong; Beyond3D is not only a beta partner of FM but they are very much publically associated with FM and if they feel FM is doing something 'not right' I would assume they would want to make sure that the community knew they disagreed with it.

[Dave Baumann]

As I said before, if game developers don't care about this, then why should Futuremark? Or even worse: why should you?

Because this has always been painted as a vendor neutral, DirectX benchmark - that is what we thought we were dealing wt when we signed up to the Beta program and we made our suggestsions according to these guidelines. Somewhere between the last Q&A and the release of the benchmark that changed.

What makes you so sure that FM doesn't have that commitment from a second vendor?

Regardless of whether they do or not, that would still be a solution that has no controls in the API.

And even if they don't, who are you to tell them to stick to guidelines that they made years ago, which simply don't apply anymore?

We're a member of their beta program. We signed up to that operating under certain guilines and communications that had been made to us previously. Somewhere these changed and they weren't communicated to us.

[Scali]

Originally Posted by DaveBaumann

Regardless of whether they do or not, that would still be a solution that has no controls in the API.

The same goes for 3Dc at this time.

We're a member of their beta program. We signed up to that operating under certain guilines and communications that had been made to us previously. Somewhere these changed and they weren't communicated to us.

My point exactly, I believe the only right you have as a beta member is to quit their program if you don't like it, like NVIDIA did.
It's not like "I join the beta program and now FM has to do everything I say". They have the last vote in everything, you're free to have complaints about their choices, but they're free to ignore your complaints.