Will next generation h/w be alot harder to review?

g__day

Regular
I think there are very many websites that do very techincal, in-depth reviews of 3d cards that are poor really. They don't know what they are on about in enough depth (Kyle, Tom and Aand spring to mind at times).

But it got me wondering if NV40 and R420 come out with true VS3.0 / PS3.0 support will it make reviewing them simpler (doubt it) or a whole lot more challenging for all 3d reviewers? Will the compelixty and power of these cards make finding a good review benchmark very difficult - if indeed any software exists to even demonstrate - let alone benchmark what they are truly capable of!

What are your thoghts folk?
 
I think we should stop doing reviews on hardware feature and do it on the drivers as 99% of the time the drivers don't support the supposed hardware features. Lets look at D3D on nvidia cards and trilinear filters well its not their even though the hardware can do trilinear. SSAA is probably never gonna be enable on R3XX drivers for the PC and well GLSlang is here and there still isn't a F-buffer which was so strongly touted at the launch of the R350.

Now there isn't the software yet out for PS 3.0 of VS 3.0 but I'm sure humus will do us some demos which we could use for benching :) I don't know how much of a big deal PS 3.0 will be I believe it is mainly going to be the greater capablity limits with some other features. VS 3.0 is gonna be a reasonable increase in features but still I think for most application if the features are full HW accelerated they aren't gonna be the bottleneck.
 
Oh, I'm sure some of the smarties here will come up with new programs for B3D to test.

Of course, it will be the same thing again wrt actual games featuring shders_3_0, which is to say probably none at time of HW debut.

And then there's the next 3DMark, of course...
 
I still think games will be the most important piece of software to test a video card with.

Not sayin other applications aren't usefull for certain things though.
 
Well, for people that aren't so tech-savvy, simple performance comparisons should be enough.

But yes, the inner workings will be very challenging to discern. In time, things will get more simple, as enthusiasts and technology reviewers get more used to the things to look for (since architectural change is going to be slowing down), but until then, we'll have to deal with not really knowing what an architecture is capable of, and what its problems really are, until months after release.
 
Brent said:
I still think games will be the most important piece of software to test a video card with.

Not sayin other applications aren't usefull for certain things though.
Yes, synthetic benchmarks are for testing potential. They're for examining specific aspects of a video card's performance. They are of some interest to enthusiasts, but their real benefit is for developers, because through such benchmarks, developers can see where to focus their engine design.

That's why I despise 3DMark. It tries to be a game benchmark.

But definitely, for those of us that actually play games, game benchmarks are the most important.
 
It'll be nigh-impossible with regards to new features for about 3 months. By the time the mainstream cards come out, apps will have come out that really test the new features.
 
well if Futuremark can release something much like 3dmark2003 in which it tests only the graphics card.. I'll like it. I think they did a very good job with 3dmark2003 and all the people complaining about CPU limitisations should go cry again.

keep up the good work Futuremark and Beyond3D.

US
 
3DMark is a terrible benchmark. It tries far too much to be a game benchmark, but the goals are entirely different from a real game. This makes it irrelevant as a benchmark of performance within games.

A good synthetic benchmark is one that tests specific aspects of a video card. The 3DMark score does not do this, and the specific tests that 3DMark does offer are pretty narrow in scope, and thus not very useful.
 
Chalnoth said:
3DMark is a terrible benchmark. It tries far too much to be a game benchmark, but the goals are entirely different from a real game. This makes it irrelevant as a benchmark of performance within games.

A good synthetic benchmark is one that tests specific aspects of a video card. The 3DMark score does not do this, and the specific tests that 3DMark does offer are pretty narrow in scope, and thus not very useful.

Please, let's not start this again, ok?

Mods, if this turns into another 3Dmark is useless/terrible discussion please lock immediately.

Tommy McClain
 
AzBat said:
Please, let's not start this again, ok?

Mods, if this turns into another 3Dmark is useless/terrible discussion please lock immediately.

Tommy McClain

Sir, yes sir! My cursor is now locked into position above the nuke thread button. 8)
 
In any new benchmarks I'd like to see shadow performance tests. Right now both ATI and NV's top cards seem to perform quite badly in this area.
 
THe_KELRaTH said:
In any new benchmarks I'd like to see shadow performance tests. Right now both ATI and NV's top cards seem to perform quite badly in this area.
One problem with this is that using the stencil buffer for shadowing is going out, and the products from the two companies, apparently, don't have a common shadowing technique between them other than stencil shadowing. They do, however, both support a method of shadowing that is superior in performance to shadow volumes (nVidia's is the more robust method right now).

So, what method would you choose to do the shadowing? Each has its drawbacks, and not all are in visual quality and performance. Shadow volumes are going to be a flash in the pan, as other, faster shadowing techniques will take over.
 
Chalnoth said:
3DMark is a terrible benchmark. It tries far too much to be a game benchmark, but the goals are entirely different from a real game. This makes it irrelevant as a benchmark of performance within games.

Of course its goals are different - its a benchmark, not a game.
What's your point?
It is still relevant to critisize the instances where it doesn't use optimally efficient code because it invites/simplifies cheating. But it doesn't otherwise make the data any more or less relevant. Which ties into....

A good synthetic benchmark is one that tests specific aspects of a video card.

I'm sorry to be blunt, but this statement is complete and utter nonsense.
The quality of a benchmark is directly tied to how feasible it is to apply its results to predicting application performance.
That the benchmark is synthetic simply doesn't enter into it, other than that it can be designed not to use very specific rendering tricks that you could predict would introduce corresponding bias in the results. That's an advantage of synthetic benchmarks mimicking real applications.

Take SPEC (int/fpu) as an example. The benchmark codes are selected to be representative for the application space to be modeled, to have reasonable memory access characteristics et cetera. The care with which the SPEC suite is constructed makes it as good a predictor as is possible for its target application space. Pretty much unarguably better than just about any real application(s) except of course if you can benchmark exactly the ones you are interested in with exactly the same parameters that you will use for production runs. Generally not a realistic option these days. (Yes, the SPEC codes have their origin in actual problems. Doesn't make the suite any less artificially constructed. Constructed=better, if done well.)

Testing specific aspects is possible, but the problem of going from the data points to a prediction about application performance becomes difficult or impossible. Difficult but doable for developers - impossible for consumers/reviewers who do not know the exact specifics of how the games are coded.

There has been one very important exception to this and that is the parameter that is measured as "fillrate" in the 3DMark tests. I did some lightweight multivariate statistical analysis three years ago which indicated that out of all the synthetic tests 3DMark used, the fillrate numbers were the only ones that correlated with games performance at all. The rest were deep in the noise. The domination of fillrate for games was also the reason that all those pages that reviewers filled with performance graphs looked remarkably similar - they were effectively showing fillrate graphs over and over and over again. They might as well have run Q3 at high resolution (to measure effective fillrates) and low resolution (to be able to factor out the host system performance) and be done with it.

The 3DMark score does not do this, and the specific tests that 3DMark does offer are pretty narrow in scope, and thus not very useful.

They never were useful, apart from fillrate. Interesting - yes, useful - no. They were always there to satisfy curiosity, basically.
Again, subsystem measurement requires a subsequent synthesis step in order to be able to make application predictions. Which is nigh on impossible to do if you don't have the code in front of your eyes, and not particularly trivial even then.

Benchmarking, when done for utilitarian purposes rather than to alleviate boredom, is all about prediction. Judging by what little data we have available to us, this task will get somewhat more difficult in the future because fillrate might not be the one overwhelming factor. This does not in any way imply that synthetic benchmarking is worse off than application benchmarking. Look at the datapoints in that ugly little benchmarking paper that came out of ATI - does it look as if one application would be a particularly good descriptor of the DX9 group? The approach taken both by SPEC and FutureMark - to include tests that are representative of different application behaviour, and then weighing the results to a final score while still requiring that individual subresults be published, is very reasonable. You might quibble with the weighting scheme. You might argue that the precision in the predictions made possible is not good enough for the problem you want to adress (this may well be true!). But the approach is sound, and is as good as anyone has been able to come up with in the area of general purpose benchmarking tools so far.

PS. Going away over Christmas in less than an hour, or so my wife tells me. :) Be well, all of you.
 
Chalnoth said:
They do, however, both support a method of shadowing that is superior in performance to shadow volumes (nVidia's is the more robust method right now).
I think 'robust' is completely the wrong word to use here - our support for shadow buffers, for example, is completely robust as it is a generalised and completely programmable method through a pipeline with strictly defined behaviour.

Hardwired, unsupported and unspecified in the API does not equate to 'robust'.
 
Chalnoth said:
One problem with this is that using the stencil buffer for shadowing is going out, and the products from the two companies, apparently, don't have a common shadowing technique between them other than stencil shadowing. They do, however, both support a method of shadowing that is superior in performance to shadow volumes (nVidia's is the more robust method right now).

So, what method would you choose to do the shadowing? Each has its drawbacks, and not all are in visual quality and performance. Shadow volumes are going to be a flash in the pan, as other, faster shadowing techniques will take over.

What softshadow method is the most common right now and is likely to be in the near future?
It seems that quite a number of games are implementing softshadows but is a real performance killer - well all except Rallisport Challenge it would seem.
Or is the problem that programmers of such games are just not using any intelligence and supporting a shadow technique more appropriate for current card performance.
 
andypski said:
Chalnoth said:
They do, however, both support a method of shadowing that is superior in performance to shadow volumes (nVidia's is the more robust method right now).
I think 'robust' is completely the wrong word to use here - our support for shadow buffers, for example, is completely robust as it is a generalised and completely programmable method through a pipeline with strictly defined behaviour.

Hardwired, unsupported and unspecified in the API does not equate to 'robust'.
I thought that the Radeon cards used a shadow projectors technique? I guess I was working off of old knowledge...

nVidia has been using a shadow buffer technique since the NV2x. My statement probably applied to the NV2x vs. Radeon 8500, not current cards. Anyway, do you know the differences in the way the NV3x and the R3xx support shadow buffers? I've not seen much about it.
 
Chalnoth said:
I thought that the Radeon cards used a shadow projectors technique? I guess I was working off of old knowledge...

nVidia has been using a shadow buffer technique since the NV2x. My statement probably applied to the NV2x vs. Radeon 8500, not current cards. Anyway, do you know the differences in the way the NV3x and the R3xx support shadow buffers? I've not seen much about it.
Essentially the method of support is similar - you render the depth from the point of view light source into a texture map and then use it as a source texture in the pixel shader - you do a transform of the current reference depth into the reference frame of the light and if the pixel you are rasterizing is further from the light source than the value stored in the texture then the pixel is in shadow.

My understanding is that the NV2x/NV3x have hardwired support for applying a percentage closer filter (PCF) to the depth comparison (ie. the depth comparisons are performed in a specialised unit outside of the pixel shader in the texture filtering hardware which reports back the proportion of the samples that pass/fail the depth test).

This form of filtering is not exposed in DirectX, so AFAIK it is exposed by the use of depth buffer formats as texture sources - when the driver detects that a depth texture is attached as a texture source it uses PCF filtering instead of Bilinear - this is unspecified behaviour from the API point of view since D3D has no concept of PCF.

When using older pixel shaders (<= 1.3) you also have to follow a certain specific setup to enable shadow buffering (by having a particular format for the pixel shader, and attaching the depth texture to a particular texture stage). This behaviour is also outside the strict specification of the pixel shader API for these shader versions, and as such could potentially cause compatibility issues with other hardware, even though that hardware might be implementing the spec completely correctly.

R300 handles depth textures in a completely orthogonal fashion to other formats - in fact the semantic that you are using a depth texture is entirely created by the pixel shader. You can generate the depth information into a buffer of any renderable format, and an implementation of any specific desired filtering schemes can then performed by using a pixel shader, although naturally the more complex the filtering scheme the more pixel shader performance is required. In fact R2xx can also support shadow buffering through PS1.4 - there is an example application demonstrating this on the ATI developer website - the available depth-resolution of the shadow buffer is rather more limited on R2xx than on R3xx which can use floating point texture depths.

Of course it should also possible to use the same shader method on NV3x hardware, although currently the available depth resolution might be lower because floating point render targets are not available.
 
andypski said:
Of course it should also possible to use the same shader method on NV3x hardware, although currently the available depth resolution might be lower because floating point render targets are not available.
Why would you need FP render targets for shadow buffering?

btw, I think it's a poor design decision that "D3D has no concept of PCF". Same for the ARB shadows extension.
Orthogonality is desirable, yes, but the cost is sometimes higher than what you gain. Not to mention that it's very easy to include in the API.
How many PS instruction do you need to do PCF? Four tex, a cmp and a dp4, and it takes four 2D interpolants (or two and an add alternatively)? One tex seems to be the better choice.
 
Back
Top