Pixel shader benchmarks

pcchen

Moderator
Moderator
Veteran
Supporter
I am working on a pixel shader benchmark framework. Basically it let you assign your own shaders, textures, texture coordinates, etc. for testing the pixel shader performance of a video card. It also has a "precision tester" for testing the mantissa bits of the pixel shader.

Now I need some opinions :) Currently this program renders the same frame for many times. Should I change this (make it rotating etc) to avoid possible cheating?
 
Hmm... there are a few other things about anti-cheating. For example, maybe I should copy and save one frame in the middle of the test for checking. However, there are always some "workaround" for these. Perhaps I should just bet that IHVs won't bother cheating against my program :p
 
pcchen said:
I am working on a pixel shader benchmark framework. Basically it let you assign your own shaders, textures, texture coordinates, etc. for testing the pixel shader performance of a video card. It also has a "precision tester" for testing the mantissa bits of the pixel shader.

Now I need some opinions :) Currently this program renders the same frame for many times. Should I change this (make it rotating etc) to avoid possible cheating?

i think those things (bolded in your original posting) are pretty good anti-cheating factors - an arbitrary shader, texture and texmapping would make cheating very tough.

re rotation of the scene - you could do that, as it could prevent any cheating based on the premise that all the view settings are constant throughout the test (anon IHV: hey, let's cache the window coordinates and spare us the projection work altogether). OTH, though, you yourself may want for the transformation part of the pipe to affect as least as possible the resulting "score" of the test as it is a _pixel_shaders_ test after all (you'd be best with screen space geometry). but it's unlikely that _all_ IHVs would follow that test-beneficient "optimization" as that one fictitious anon IHV above :)

another concern re rotation (or any other arbitrary transform) is it could take away from the stressing of the PS test by overall decreasing the visible shaded area for the duration of the test. i.e. i imagine you have a full-view head-on polygon featuring the respective shader and texturing - if you rotate that you may end up with less test-relevant area for the duration of the test session. instead of full rotations in time, you may go for one special rotation (or another transform) at a single, random frame of the test, at which time through some form of feedback you'd check whether the transform actually took place. that would show whether the drivers honor the transform pipeline or they "optimize" it out.
 
Thanks for your comments. :) Regarding to rotation, I am going to rotate texture coordinates only. So the polygon still fills the entire screen.

I am more worry about discarding frames. Since my program renders only two triangles (a big quad) with very few state changes, a "malicious" driver could buffer them and try to discard few frames. Rotating makes it easier to spot such cheats but it is still very hard to notice when frame rate is very high (with very short shaders it can be several hundreds per second).
 
Try timing the time between flips, if you get a large periodic deviation (or even a large random one for that matter) you know something is amis. Of course this may not be helped by the fact that certain IHV's may stretch the WHQL rules somewhat with regard to when you're allowed to issue a flip*.

Inspite of all the discussions going on, generally I doubt that any IHV's are currently looking for fixed "sequences" and optimising for them in the drivers. The only thing I would suggest (if you really think cheating is going to be an issue) is that you might want to add some randon noise to the lsb's of your textures when you load them and/or load them in a random order.

*WHQL states that you shouldn't issue a flip for a surface that already has one outstanding on it. UT2003 locks the render surfaces after every render as it was found that many cards/drivers do not appear to follow this rule and where buffering many frames up resulting in unacceptable lag.

John.
 
Safest way to detect skipped frames is of course to measure them on the VGA cable. And that restarted my bad concious about yet another project of mine that has been halted - the FPSdongle. I have the microcontroller software finished, and the HW design ready, except the (rather) high speed comparators that is needed to digitalize the VGA signal. I couldn't find anything cheap, simple and fast enough when I was looking at it.

Realy off-topic request: :)
Isn't there a lot of EEs here? Is there anyone that can give me a hint on a comparator? It should detect swings from ~0V to ~0.7V within maximum 100ns, and preferably be driven by a single supply (GND and 5V). I need 3 of them, so it should preferably come in a dual or quad package.


The end result when the tested software supports it, is a file with timestamps for every single frame. Both when the flip was done, and when the frame hits the screen. With a precision of ~10us. :)


But if this is too complex (which it is :)), then you could make dropped frames more visible in other ways. Moving the scene is a god start. And since it's difficult to see dropped frames at high framerates, make sure that the framerate is low. This can be done by rendering lots of screen sized quads back to front on each frame. It will give a too high score for deferred renderers, but I doubt any IMRs do optimizations that goes around it. (And right now there aren't any deferred renderers with PS to test.)
 
Hmm... since the shader is user defined, so I can't make them faster or slower. A way to make frame rate slower is to use alpha blending (like the 3DMark's fillrate tests). However, alpha blending can stress memory bandwidth. I think I'll make it an option for the user.
 
You can't make the shader faster or slower, but you could run it enough times per frame that the framerate is slow. Alpha blending is an effective way to force the PS to run for all surfaces, even for chips like PowerVR (if they had PS). But it does have the implications you said. I believe that simply rendering the surfaces back to front, or even better with z-buffer turned off, would be enough to make all chips (except PowerVR) do all the calculations for all surfaces.
 
I don't know what happened to my typing skills lately. I've recently got a new computer, and more importantly in this respect, an new keyboard. The new feeling in the keyboard could explain the *loads* of missing and duplicated letters (that I hopefuly edited out before posting). But I can't explain all the comlpletely off keys.

No I didn't misspell on purpose, and no 'w' and 'v' aren't close to each other on a Swedish keyboard.
 
:LOL: Actually I have seen some official emails go out with the same mistake, it just always surprises me since those keys are soo far apart. Ah well...

K-
 
Basic said:
Safest way to detect skipped frames is of course to measure them on the VGA cable. And that restarted my bad concious about yet another project of mine that has been halted - the FPSdongle. I have the microcontroller software finished, and the HW design ready, except the (rather) high speed comparators that is needed to digitalize the VGA signal. I couldn't find anything cheap, simple and fast enough when I was looking at it.

Not sure how that helps you, if the app is repeatedly rendering the same frame and the drivers decide to skip some of them, the you just end up with a previousley rendered frame emmited it looks the same both visually and electrically...

Are you looking for a change in frame content in your 'dongle' ?

Realy off-topic request:
Isn't there a lot of EEs here? Is there anyone that can give me a hint on a comparator? It should detect swings from ~0V to ~0.7V within maximum 100ns, and preferably be driven by a single supply (GND and 5V). I need 3 of them, so it should preferably come in a dual or quad package.
Just look for a high speed op amp, particularily those targeted at video apps these are generally available in dual or quad packages. Although you might have fun stopping the things from oscillating....

John.
[/quote]
 
The app must support the dongle by drawing a line at the left edge of the screen. The line should have a new color each frame, and that's what the dongle detects. With a realy fast comparator, it could be just a 2-pixel wide line. If the comparator is slower, the line have to be wider. But it still should have a minimal effect on framerate.

I'll take a second look for an OP-amp. But OPs for video sounds like it can be expensive (I want this to be realy cheap, since it honestly doesn't fill a very important function. :))

More specifically, the dongle sample the red and green channel (one bit each) at the first positive edge on the blue channel after a hSync. So you get a 2-bit tag on each frame. A timestamp is recorded whenever this tag (or some other things) change. Realy dirt simple.
 
If you make it the whole top or bottom line then you have uS to sample in..
Aren't you going to have the issue of needing to 'holed' the analogue values for compraison from frame to frame ? Sounds like you need to take a sample digitally, if you use a whole line and only look for large swings you could probably get away with a simple digital intergrator i.e. just feed the rgb signals into gate and periodically sample and inc as necessary. Hmm but this would be constrianed by refresh rate...
 
Going back to original topic...

Just use multiple translucent layers to get the frame rate down, this removes any early Z or deferred rendering impact when BM'ing the PS..

[Edit] Forgot to add that you remove the impact of BW by making sure you're marginally shader bound, guess the prob with that is that you're relying in the person writing the test shader to understand it...

John.
 
Now I have the first public beta uploaded. Unfortunately, I have no time to work on an English document yet, so currently the document is in traditional Chinese only. However, there are some comments inside the setting file which may be useful.

Download

Simple usage:

fillrate9 [setup file] [output file]

setup file: default to config.txt
output file: default to output.txt

File names can not contain any space character.

I put a setup file called "precision.txt" which can be used for testing the mantissa bits of pixel shader 2.0. On my Radeon 9700 it's 16 bits.

I found some interesting things about R300 when writing this tool. I find out that "abs" instruction is essentially free on a R300 when optimizing the precision checking pixel shader. I also did some experiments about the normalization instruction (nrm) and I think it's a 3 cycles instruction. I believe one can find more interesting things using this tool. :)
 
It requires DirectX 9 installed and a 3D card with pixel shader 1.1 or later support. Pixel shader 2.0 precision test requires pixel shader 2.0 support.

It is developed with a Radeon 9700, but it was tested with GeForce 4 Ti and GeForce FX 5200.

There are some prewritten setup files and shader files. config.txt is a pixel shader 2.0 test (the shader file is simple.pss). config11.txt is a pixel shader 1.1 test (shader file is simple11.pss). Cards support only pixel shader 1.1 should use config11.txt or write your own pixel shader.
 
Back
Top