New D3D RightMark beta (Rightmark3D DX9 synthetics)

[Philipp S. Gerasimov]

Hi everyone!

You can find the new beta of D3D Rightmark here:

Public Pre-Beta 2 ~5.5 mb

Changes/Fixes:

- new ScreenShot tab in all modules
- new shaders for Pixel Shading test (now HLSL) (use Det44.65+ for 16fp/32fp comparison)
- more sophisticated log file (Exel 2002 only)...
- you can stop the batch by pressing ESC.
- new State Profiler test (very early version)

 

[demalion]

The state change analysis is something that I have been looking forward to seeing used. I really hope it is a primary focus for growth.

Might I also recomend some further system bus analysis functionality as another uniquely useful feature? Seems extremely relevant with a bus migration coming up that directly relates to the 3D industry. Also, competent analysis of this factor introduces another focus for IHV attention, which should hopefully be a good thing...which brings to mind the question of whether you are considering the impact of application detection and how you plan to effect such a mechanism?

The first thoughts that came to mind...I'll play with it later.
EDIT: Well, with my new, faster system, I decided to play with it now instead. I think it this bechmark is shaping up quite well...very nice execution, and useful presentation, though the Excel 2002 dependency does seem odd (why no CSV, and why does the xml usage require Excel 2002?).

 

[Philipp S. Gerasimov]

demalion

though the Excel 2002 dependency does seem odd
Our video editors use Excel for diagram creation for reviews...
But we can add support for HTML logs, etc.

which brings to mind the question of whether you are considering the impact of application detection and how you plan to effect such a mechanism?
Application detection by drivers? We have great relationships with nVidia and ATI DevRels, so we do not expect this...

 

[demalion]

demalion

though the Excel 2002 dependency does seem odd
Our video editors use Excel for diagram creation for reviews...
But we can add support for HTML logs, etc.

Please do...more options are better. But Excel can handle CVS, as can other spreadsheets and database programs. I just found it odd that you specifically required Excel 2002 format.

which brings to mind the question of whether you are considering the impact of application detection and how you plan to effect such a mechanism?
Application detection by drivers? We have great relationships with nVidia and ATI DevRels, so we do not expect this...

Hmm...well, I don't think your relationship includes controlling what they do, and your expectation in this regard sounds like trouble. Actually, it sounds like a really really big mistake not to maintain a level of cognizance about what is going on wrt to your benchmark, as it isn't inherently immune to abuse.

Aside from what I can only see as, I apologize if I sound harsh, and exceedingly foolish amount of trust and choice to ignore what's already happened, I can only begin to guess as to why you don't expect this.

If you expect your benchmark to become popular, I can hardly begin to guess why you expect not to be targetted when other benchmarks are.

As far as ATI, maybe you trust their actions based on what they've done, what they've not done, and how they reacted to things that were said...that does make some sense because of the particular actions they've taken, but it also a disservice (in principle) to the users of your benchmark for depending on ATI's integrity and not your own cognizance.

As far as nVidia, the only way it seems you can trust the company at all is if you let them dictate a custom codepath to you for benchmarking...devrel doesn't run the company (unfortunately, I think) and I think you should consider that in light of whatever relationship you have with them. That said, maybe you do plan to provide that custom code path...the name of the program (D3D Rightmark) has caused me to think that you might have a "Cg Rightmark" (as well as an eventual "OpenGL Rightmark") in the works, especially as the initial program required Cg. Perhaps that is why you expect nVidia not to do application targetting...but that depends on nVidia management's willingness to let the benchmark users have a choice in the matter (you plan on adding choices, not taking them away, right? That's directly against nVidia's stated intentions for benchmarking).
If it becomes popular, what makes you think they will behave differently than they already have with benchmarks...or do you simply not expect Rightmark 3D to become popular?

It sounds to me like you are mistaking the devrel comments and relationship for assurances of the future actions of the people who make the decisions. Where is there an indication that this is realistic at all? That's a serious question...perhaps you have some information you could share.

Then again, if you're not charging money for it, perhaps this level of cognizance (from the original authors, atleast) is asking too much.

 

[vrecan]

demalion
Verry well put. I agree completly. Just because you have good relations with their devrel team doesn't mean that their management wont see your benchmark as a threat if they aren't scoring the way they want. Seems kindof errogant now days to build a benchmark that you want to become popular yet put no effort towards "optimization" detection. This isn't targeted just at nvidia no company likes a benchmark that makes their product look bad.

 

[cho]

most benchmark even real application would/will be detection by drivers, it is hard to avoid .

the 3dmark03 Bulid 330 have tried to anti-detection , but now the 44.65 w/bulid 330 had got almost same result(include renamed AF8X) and better image quality(PS 2.0 test suit) than the 44.03 with 3dmark 320.

self/random re-name the file name and the shader names ?? i think both ati and nvidia also have way to detection such thing .

 

[pcchen]

The best way is to randomly change shaders.Swapping registers is a good start (swapping r0 and r1, etc.). Swapping some instructions are better. However, this may make the benchmark results inconsistent. Furthermore, if they really want to, it still can be detected.

 

[Philipp S. Gerasimov]

demalion
Please do...more options are better
ok

or do you simply not expect Rightmark 3D to become popular?
We just want to create helpful tool for video reviewers, for some game enthusiasts, but not "the PR weapon"... synthetic benchmark, popular or not, is only addition for game tests.

And many of our results are bounded by hardware specs...

 

[Hellbinder]

And many of our results are bounded by hardware specs...

Thats an interesting commnet...

 

[demalion]

demalion
Please do...more options are better
ok

Thanks!

or do you simply not expect Rightmark 3D to become popular?
We just want to create helpful tool for video reviewers, for some game enthusiasts, but not "the PR weapon"... synthetic benchmark, popular or not, is only addition for game tests.

Yes, of course they are in addition, but they still have to represent something distinct and useful when that is done. Application detection circumvents that usefulness, and it can easily be in an IHV's interest regardless of whether games are used as well, or not.

Observe that games are also targetted in ways that don't serve the consumer...fps gains at significant reduction in image quality and by ignoring or circumventing user specified settings are done to games as well when they are used as "game tests". The control panel option to reduce image quality serves the user...when control panels are overriden, it does not, and the same with precision specifications. It is the latter cases we are discussing, and being used in addition to games does not prevent your benchmark from being a target for that

Now, this is a disservice for both games and benchmarks, but the benchmark app is more suited for providing detailed exposure of strengths and weaknesses. The focus on details is unique to benchmarks, but this can also provide a simpler environment for misrepresentation. If your benchmark tests details that it would serve an IHV to misrepresent, why would game tests also being used prevent that happening when they don't expose those details as clearly?

By the way, this is something I've talked about in detail before, if you have the time to search on my name using terms that you think might be relevant. I'd recommend "3dmark", "rightmark", and "open AND source AND benchmark"

And many of our results are bounded by hardware specs...

I think I know what you are saying...that your tests are simple and direct and not prone to circumvention? Well, you don't think there are opportunities to exploit the predictability of what you test to misrepresent that? Also, do you intend to never to add new tests? I'm not saying you should not do things that can be circumvented amd distored, I'm asking you to recognize and consider that the things you are doing to measure fps results from your tests already can be, and what can be done to address that.

 

[Ante P]

The state change analysis is something that I have been looking forward to seeing used. I really hope it is a primary focus for growth.

Now I'm going out on a limb making myself look stupid here, but..
What exactly does it test?
I mean I can see by the config that it will change shaders, textures etc. But more exactly what does it test?


BTW great work on the suite guys. I've tried the earlier versions and they were pretty much a mess IMHO but this one seems stable and easy to use.
It's also thankfull that the tests don't run "forever" like they did in the first versions.

Oh yeah, I get no difference at all when chosing between FP16 and FP32 is the PS tests on a 5800. Haven't tried the 5900 yet but I assume it will show a difference?
(That is if FP32 is still "disabled" on all sub-5900 boards in the 44.65 drivers?)

 

[cho]

44.65's render target format still not support fp16/fp32, but 5x.xx will support such format.

 

[Philipp S. Gerasimov]

Ante P
BTW great work on the suite guys
Thanks!

I mean I can see by the config that it will change shaders, textures etc. But more exactly what does it test?

ChangeStates
ChangePShader
ChangeVShader
ChangeTextures
ChangeVBuffer

for NumChanges
{
if( ChangeStates ) ChangeStates
if( ChangePShader ) ChangePShader
if( ChangeVShader) ChangeVShader
if( ChangeTextures) ChangeTextures
if( ChangeVBuffer) ChangeVBuffer

RenderNTriangles
}

Oh yeah, I get no difference at all when chosing between FP16 and FP32 is the PS tests on a 5800.
Hmm... Looks like a bug. Maybe there are problems with DX9a (I use DX9). We has seen difference on most GFFX... This is not 2x, but 10-30% on GF58000U. ( 5th shader, window: 49.3 / 66.5 )

And please turn OFF AA and AF with control panel...

 

[Ante P]

Thanks Philipp.
I must admit though that the explanation didn't make me much wiser. What does it more practically mean? (If you can't be bothered by my noob questions it's ok )

I just tried the fifth test and here I do see a difference between FP32 and FP16, about the same as you stated.

I forgot to mention one problem though. I get severe stuttering in all the tests on a 5800. Never had such problems in any other test/game so I dunno what's up with it. fps is high but it still stutters (rather pauses almost) like craxy.

 

[demalion]

Let me offer my understanding, with following corrections giving you some opportunities for further insight.

The issue is how the driver communicate with the GPU. It doesn't just send the textures, shaders, geometry, etc....it signals the GPU about what it is doing, and the GPU gets ready to receive it. My simplified way of describing a "state" is simply as the GPU being ready to receive a specific type of data, or of being in the middle carrying out a particular action on the data it has received, or of being told to send back data to the system.

I can imagine all sorts of complexities and modifications to prevent the impact of the worst case for different designs...but for simplicity I'll just offer that switching these things around when not necessary would likely tend to slow things down on the GPU and/or make the AGP bus a bottleneck when it didn't have to be.

Hopefully some interesting details will be forthcoming in any further input that is offered to this simplified (due to lack of detailed familiarity) explanation, and we'll both learn something.

 

[Philipp S. Gerasimov]

demalion

Okey. With "states" we can setup different parameters for rendering:
face culling type (ccw,cw),
primitive filling type (solid,wireframe),
alpha blending (enable/disable, params),
z-stencil testing,
setup texture stages, etc.

When my application(game) render the scene I change states, depending on current rendering method, material, model, etc... If I do very big number of changes, rendering is un-effective. For example, 1 state change ~= 100-200 rendered triangles. Same situation we have with texture changing (setuping current texture for rendering), vertex/index buffer changing, shader and shader constant changing... Of cource, the driver caches "states" and trying to optimize changing, but we still need to think about it.

So, our test trying to show "price" of changing for states, shaders, buffers...

PS. Pixel Shaders are most "greedy" for changing.

 

[demalion]

Hmm...I was a bit stuck on a driver discussion I had with someone about AGP bottlenecks. Of course, from the application's viewpoint it should be concerned with rendering control more than the data transfer emphasis I stated.

 

[Philipp S. Gerasimov]

New beta of D3D Rightmark (1.0.2.7 Public Beta 1):


D3DRightMark.zip ~6.5mb

Changes/Fixes:

- Options dialog
- HTML format for results
- new shaders for Pixel Shading test
- Source code for all test modules
- Add button (add new batch file to current batch)
- Tooltips for some interface elements

 

[Magnum PI]

please could you allow output to comma separated values ?
it can be used by *every* spreadsheet in the world..
thanks

 

[AndrewM]

Okey. With "states" we can setup different parameters for rendering:
face culling type (ccw,cw),
primitive filling type (solid,wireframe),
alpha blending (enable/disable, params),
z-stencil testing,
setup texture stages, etc.

When my application(game) render the scene I change states, depending on current rendering method, material, model, etc... If I do very big number of changes, rendering is un-effective. For example, 1 state change ~= 100-200 rendered triangles. Same situation we have with texture changing (setuping current texture for rendering), vertex/index buffer changing, shader and shader constant changing... Of cource, the driver caches "states" and trying to optimize changing, but we still need to think about it.

So, our test trying to show "price" of changing for states, shaders, buffers...

PS. Pixel Shaders are most "greedy" for changing.

I really hope that you're not just timing the function calls. This will give you virtually no indication of how the GPU is reacting to state changes. It _will_ give you how long it takes the driver to submit the rendering request to the push-buffer.