Aquamark3 Preview at 3DGPU

[stevem]

3) I have asked Massive to comment on the concerns raised here, as their MD previously stated that shaders are assigned soley by DX9 featureset detected not by performance. Its tricky reconciling what Alexander - their MD has said and Ingo Frick - their Chief Technican posted on guru3d and how we therefore compare apples vs apples if that is what we are doing!

Let me re-quote a part of his response from your earlier post. No further qualification is necessary re: AM3 benchmark numbers with 51.75.

Ingo Frick: "But you still have to pay attenantion regarding other resulting numbers. For instance, if you want to compare the number of triangles rendered per second, you compare apples and oranges because multipass techniques lead to significantly different numbers compared to multitexture techniques."

They do it because they don't really have a choice.

Exactly correct. It became apparent earlier this year...

 

[Bouncing Zabaglione Bros.]

Ingo Frick: "But you still have to pay attenantion regarding other resulting numbers. For instance, if you want to compare the number of triangles rendered per second, you compare apples and oranges because multipass techniques lead to significantly different numbers compared to multitexture techniques."

Doesn't this give you misleading numbers? If ATI need to process less polygons using PS 2.0 (and are faster because of it), surely they *should* score higher than Nvidia who have to process more polygons using PS 1.1 ?

It seems silly to rely on counting polygons as a measure of performance when one architechture can be much faster in real life applications by not having to render all those polygons in the first place, whilst still maintaining the required image quality.

What MD seem to be doing is saying "well, ATI have a turbo in their engine, but because Nvidia doesn't, we'll take the turbo out of the ATI engine to give an "apples to apples" comparison, and this will accurately tell us which car is faster".

 

[Mark]

Yes, AquaMark3 only has 4 PS2.0 shaders, but those 4 shaders are used a lot throughout the benchmark. Here is a pic I snapped while testing SVIST (Shader VISualization Test). The red areas are PS2.0, the yellow are PS1.x, and the blue area have no PS applied (the bright plume represents an area of high ovedrawn, in this case an explosion created by the particle system). Going by SVIST; the terrain, the vehicles, the buildings, and the rock formations all use PS2.0. On the ATI DX9 hardware I have here when runnig SVIST, most of the scenes are at least 50-60% "red".

 

[stevem]

Doesn't this give you misleading numbers?

Maybe...

 

[Veridian3]

Yes, AquaMark3 only has 4 PS2.0 shaders, but those 4 shaders are used a lot throughout the benchmark. Here is a pic I snapped while testing SVIST (Shader VISualization Test). The red areas are PS2.0, the yellow are PS1.x, and the blue area have no PS applied (the bright plume represents an area of high ovedrawn, in this case an explosion created by the particle system). Going by SVIST; the terrain, the vehicles, the buildings, and the rock formations all use PS2.0. On the ATI DX9 hardware I have here when runnig SVIST, most of the scenes are at least 50-60% "red".

Interestingly when you run the SVIST on an FX and a Radeon you get the same output from the program...i.e. Aquamark still thinks its using PS 2.0 in all the same places on both cards.

Stu

 

[Joe DeFuria]

while one IHV is trying to convince others that it is.

Once again it's the same as before.

No, it's not.

 

[Joe DeFuria]

Well said, though we probably aren’t going to go as far as denying coverage altogether (that’s silly really).

Actually, I don't think it's silly at all.

It will probably take something like that to get nvidia to listen.

I don't mean deny coverage forever. I mean at this point, deny coverage until the drivers that nVidia supplies to you are reasonably cleared of any wrong doings. Just change the assumption: don't assume the drivers are legit...assume they are cuorrupt.

Assign one group (B3D is the likely candidate, IMO), to investigate a driver. If no odd bheaviour is found that aren't satisfactorally explained, the review sites go ahead with reviews.

If questionable, and unanserable issues are found, no reviews.

Still, it’s no secret that it’s becoming more and more difficult to make fair comparisons with so many different factors (like app detection) becoming an issue when these things obviously shouldn’t be an issue in the first place (well, in a perfect world). Kyle at the [H] won't be hard to convince about an industry wide video benchmarking standards practice of some kind, he's very open minded (despite what some here think ).

Lol...I've been hearing some snippets of some of H's comments on this whole thing, and "open minded" is not how I'd describe him.

 

[Joe DeFuria]

Interestingly when you run the SVIST on an FX and a Radeon you get the same output from the program...i.e. Aquamark still thinks its using PS 2.0 in all the same places on both cards.

Stu

THis is one question I have concerning SVIST. I assume that if AM3 sends PS 2.0 commands / shaders to the driver, but if the driver changes it to PS 1.4 or 1.1 internally, I assume that SVIST will still report PS2.0

So SVIST, while a good idea, is easily subject to being cheated. As far as I can tell, SVIST is useful for seeing how AM3 requests each pixel to be filled....but not how each pixel is actually filled.

Does AM3 offer the end user the option to force code paths? (Force R300 to behave as a PS 1.4 card?) Alternatively, I recall there may have been some registry tweaks to force R300 to report itself as a PS 1.4 part. Would be interesting to see how that might impact quality of AM3...

 

[Bjorn]

I don't mean deny coverage forever. I mean at this point, deny coverage until the drivers that nVidia supplies to you are reasonably cleared of any wrong doings. Just change the assumption: don't assume the drivers are legit...assume they are cuorrupt.

Assign one group (B3D is the likely candidate, IMO), to investigate a driver. If no odd bheaviour is found that aren't satisfactorally explained, the review sites go ahead with reviews.

If questionable, and unanserable issues are found, no reviews.

Sounds very reasonable to me.

 

[Evildeus]

Sounds very reasonable to me.

I don't think so, at least when new products are launched, because then you just need to give the driver to a reviewer who prefers to make a scoop (ie giving first benches) than investigate the drivers.

 

[Joe DeFuria]

Sounds very reasonable to me.

I don't think so, at least when new products are launched, because then you just need to give the driver to a reviewer who prefers to make a scoop (ie giving first benches) than investigate the drivers.

Of course that's the problem. That's why we're suggesting an alliance of sorts. Get the most prolific sites together.

Then Joe Blow can review what he wants to for his own selfish reason, but the "big guys" in one unified voice can say. "Don't trust those results."

 

[Veridian3]

THis is one question I have concerning SVIST. I assume that if AM3 sends PS 2.0 commands / shaders to the driver, but if the driver changes it to PS 1.4 or 1.1 internally, I assume that SVIST will still report PS2.0

So SVIST, while a good idea, is easily subject to being cheated. As far as I can tell, SVIST is useful for seeing how AM3 requests each pixel to be filled....but not how each pixel is actually filled.

Does AM3 offer the end user the option to force code paths? (Force R300 to behave as a PS 1.4 card?) Alternatively, I recall there may have been some registry tweaks to force R300 to report itself as a PS 1.4 part. Would be interesting to see how that might impact quality of AM3...

Force code paths is not an option...well..not an option currently available through the main interface. If it can be added or hacked...i'm not sure of that.

Stu

 

[Evildeus]

I don't think alliance is needed. Making a preview/review is one thing, making an investigation on of it works is another thing. Plus, every sites has some sort of specialisation that i like and is needed (even if i don't look at all, they all have an odience). Alliance doesn't adequate with quality or objectivity.

Finally, an alliance, could be also misleading. We don't know what will happen in the future, and an alliance could cover some parts and not other ones, whatever is the reason. It is at the choice of the site to put money on their procedures, technical efforts, and sometimes differentiate their products. If they don't do that well, they will slowly perrish.

Just my 2 c.

 

[Dave Baumann]

THis is one question I have concerning SVIST. I assume that if AM3 sends PS 2.0 commands / shaders to the driver, but if the driver changes it to PS 1.4 or 1.1 internally, I assume that SVIST will still report PS2.0

It can only report what shaders are supposed to be running. It has no clue what is actually running if precompiled shaders are actually being executed.

Pre compiled shaders do not have to necessarily bear any relation to any PS/VS constructs that are allowed in DX. Pre-compiled shaders are more than likely to be machine level code, which can break down some of the limitations that DX may impose (such as the interactions between the Floating and fixed units in the FX series) and so they are able to do very different things with pre compiled shaders (which is why they are getting such large performance differences between DX/OGL assembly and pre compiled versions).

 

[Dave Baumann]

Finally, an alliance, could be also misleading.

An alliance would only take the form of agreeing not to preview a product until official, WHQL'ed drivers were available - although even there there would be issues with this.

 

[g__day]

http://arc.aquamark3.com/forum/showthread.php?s=&postid=1901#post1901

Wel I asked more about how shaders are assigned to classes of video cards and how we are meant to interpret the results across different classes of DX 7 8 and 9 video cards.

Alexander was quick to answer my first round of questions so now I have posted some more too.

am3 does not know about specific graphics cards. the engine queries the dx9 capability flags and makes the decision. if i read the forums there is often some misunderstanding when it comes to our fallbacks. it seems that some people do think the we implemented fallbacks i.e. for a fx compared to a radeon. that of course is not true.

basically it is like this:

before we implement any material we decide about the LOWEST api (dx7,dx8,dx9) which is able to create the material/effect we want to have. so i.e. you do not need a fu***** ps2 for rendering a transparent particle. as we all know even dx7 class hardware was able to render transparent particles, so the material is implemented with dx7 functionality. very simple, very fast

when we need at least a dx8 hardware with ps and/or vs functionality we decide which is the lowest/fastest vs/ps version to create the material/effect we want to have. then we implement it. if a target hardware is not able to handle this (because it is a dx7 hardware, or because it has dual but not quadtexturing etc.) we create a FALLBACK (which is i.e. a dx7 texturestagestate multipass approach) solution which creates the same material/effect.

and so on.

why do we do this? because am3 uses a real world engine. we all know that for the next 12-18 month there will be some gamers out there who do not have a nv3x or r3xx or better. we as developers are FORCED to create engines that are that flexible.

let me point out that it would cost us only some hours to make all our vs/ps dx9 class hardware only. but that would be a nutty game engine. even worse it would be a synthetic game engine upon a developer would not be able to release a game in 2004 because there are not enough customers with the correct hardware out there (however ... gfx card vendors would love this)

at the end you have to implement the fallbacks if you want or if you do not want (or if you do not tell that you know that you have to do it).

To me this is the correct approach. Two major questions should be framed:

1) How do you best compare cards that report different levels of DX 7, 8 or 9 (e.g. a GF2 Ultra, a GF3 and GF FX) and hence run different shaders?

2) Over the next 6 quarters how do folk feel the mix and percentages of shaders present in games and therefore benchmarks should change? Today it might make sense to say for a top end card in a game expect about 40 shaders and of these 20% will be DX7, 50% DX8 and 20% DX9. But in a years time will the mix may be different and top end cards may be called on to run 100 shaders. What are game developers views on this point?

 

[Enbar]

From my understanding of whql it wouldn't stop any of the major cases being discussed on b3d. Whql tests the driver to make sure it passes a bunch of tests for stability and compliance to standards.

Shader replacement and app specific optimizations would not get caught by this sort of testing.

 

[Evildeus]

Finally, an alliance, could be also misleading.

An alliance would only take the form of agreeing not to preview a product until official, WHQL'ed drivers were available - although even there there would be issues with this.

I've already talked about that in my previous post. Yes there would be plenty issues, like some not-part-of-the-alliance site making a scoop preview of the hardware. I don't think it's a good way of handling things.

The best current way, i think, is to point out (by a disclamer) that no inquiry have been made on the drivers quality, therefore the numbers could be not totally showing the truth... Even then...

*Edit* BTW 45.23, 44.03 are WHQL and there's still some issues from what i've understand, it's not the ideal solution either.

 

[WaltC]

Sounds very reasonable to me.

I don't think so, at least when new products are launched, because then you just need to give the driver to a reviewer who prefers to make a scoop (ie giving first benches) than investigate the drivers.

Well, I really think increased cooperation between sites in such a manner would be a very positive thing for the sites and the general public. I could easily get cynical about the likelihood of its success because of certain "personalities" involved with managing the content on a couple of these sites, but even without them an alliance between several others with agreements and interactions between them could be both very influential and beneficial.

One thing they could do if they could agree on it, would be to word a joint "open letter" to IHVs about WHQL drivers *and other things* which all of the sites would carry as a link on their front pages for a period of time they deemed appropriate. You could list in it a series of points you all agree are important--and then list a manufacturer's "rating" where you judge how the specific IHVs are doing, and that kind of thing. There are lots of areas in which this kind of collaboration could work and produce positive results for all of the sites involved. I would love to see increased dialogue between websites in this fashion.

The thing is the IHVs want to cater to particular web sites when possible because they recognize how that influence, steered properly, could benefit them in terms of free publicity and increased sales. An alliance of sorts between sites on matters of this nature would give those sites collectively a degree of power and influence which separately none of them have. This is the kind of situation that could result in the customer base steering the IHV, instead of the other way around--and I think that's something we'd all like to see.

One thing I'd like to see done away with entirely--that's the "exclusive" to select websites cherry picked by the manufacturer's PR machine simply for the purpose of getting slanted articles published. These are billion-dollar companies--so if they tell me they don't have the resources to put together a dozen or so new-product PR packages (when they boast about spending $400M to develop these products) to send out to a bunch of sites simultaneously I'm going to laugh at them. One other thing I'd like to see: full disclosure of any financial deals (apart from advertising--specifically meaning site ad placement and not editorial content) a hardware site has with a particular IHV (this is something I've wanted to see for a long time.)

IE, this kind of IHV "cherry-picking" among web sites has been going on a long time--under the table deals--agreements to "keep mum" about certain things with promises of future product *exclusivity*--and so forth. There is much that happens behind the scenes. Collectively, the hardware sites could run the show pretty much. Until that happens, however, the manufacturing companies will seek to keep them divided and fragmented and sometimes at each others throats, because then they can continue manipulating some of them. That's what needs to be recognized and stopped--there is simply no sense in web sites spending their own time and money to function gratis as PR-extensions of certain corporations. The web sites are in the driver's seat--but the IHVs (well, not always all of them) will do their best to convince them otherwise.

I think the best reason for sites to consider something like this is because what's the sense in having influence in a market if you don't influence it for the better?

 

[g__day]

There are two distinct topics running in this thread - one related to the topic of Aquamark 3 being previewed at Guru3D - and one about controls and practices to make it easy to avoid, spot and deal with sneaky drivers being dumped on overloaded reviewers.

Both are great topics, but could we start a thread that deals with that second conversation and get all the posts in here that are actually related to that lovely topic moved into its own thread please! 8)