Benchmarking mode

[Humus]

I have implemented a way to record your own flyby paths in my framework and a way to play them back. This way all my demos can be used as benchmarks. What's left to be done now is log framerates etc. So I wanted to ask in what format it's most desireable to have the output in and what information you want to be available there, fps, time/frame, etc?

Also, demos would have to be recompiled for this to work though, so what demos would you like to be recompiled with benchmarking support?

 

[chavvdarrr]

CVS should be OK for output.

Edit: I was thinking about CSV - Comma Separated Values ...

 

[madshi]

Sounds great!

Hmmm... Just recompiling is not much work, or is it? The more demos you offer with benchmarking the more difficult it is for NVidia to chea... ehm... optimize in your demos. If you ask me, recompile them all. Then the reviewers can each pick those they like most.

Do you have plans to add some kind of cheat checks? E.g. can you check the shader precision?

Thank you! Your work is very much welcome!

 

[Humus]

CVS should be OK for output.

Which in this context means .. ?

 

[Humus]

Sounds great!

Hmmm... Just recompiling is not much work, or is it? The more demos you offer with benchmarking the more difficult it is for NVidia to chea... ehm... optimize in your demos. If you ask me, recompile them all. Then the reviewers can each pick those they like most.

Do you have plans to add some kind of cheat checks? E.g. can you check the shader precision?

Thank you! Your work is very much welcome!

Well, not much work for each demo, but I have quite a few. So recompiling all of them could take some time. Haven't planned any cheat checks though.

 

[madshi]

Haven't planned any cheat checks though.

Hmmm... How about storing a screenshot of a specified frame? That would be important so that one can at least manually compare the IQ. What do you think?

Besides: Do you think that your FP demos will show a noticable IQ difference between 16/24/32bit precision? We know that the fractal does. How about the other demos?

 

[Humus]

Well, there's always the F9 button in case you want to snap a screenshot. But a screenshot is not much of a guarantee that there's no cheat. The driver can run everything in partial precision but go into full precision whenever a screenshot is taken.

I don't think precision will matter much on most of my demos though.

 

[madshi]

Well, there's always the F9 button in case you want to snap a screenshot.

But it's difficult to get a specific frame this way. How can you compare the IQ of different graphics cards, when you don't have screenshots of exactly the same frame?

But a screenshot is not much of a guarantee that there's no cheat. The driver can run everything in partial precision but go into full precision whenever a screenshot is taken.

Isn't that exactly a good reason to implement your own screenshot function? Or would the driver be able to detect that, too? A start parameter like e.g. "HumusDemo.exe /screenshot=77" to save frame 77 to a bitmap file would be good enough, I think. Don't you think it would be worth it?

 

[Humus]

You can always set the exact position. That functionality has been in my framework for a long time. Try this on the console:

> pos
(338.399780, 79.666359, -58.320641)
> angles
(0.459000, 1.062000, 0.000000)
> setpos 338.399780, 79.666359, -58.320641
Command OK
> setangles 0.459000, 1.062000, 0.000000
Command OK


As for the screenshot, there's just no way you can fool the driver. You're asking the driver for the screendata. You can't read the framebuffer directly.

 

[madshi]

You can always set the exact position. That functionality has been in my framework for a long time. Try this on the console:

Oh, that's fine!

As for the screenshot, there's just no way you can fool the driver. You're asking the driver for the screendata. You can't read the framebuffer directly.

Hmmm... I remember when I did some DirectX programming (in 2D, though), there was something like a DirectSurface->Lock call, which gave me a direct pointer to read from. I guess that's not available in 3D rendering? Haven't done any 3D stuff myself yet...

 

[Humus]

The thing with locks is once again that it's a call to the driver. The pointer returned need not be pointing directly to the framebuffer, actually most likely not, but rather to a copy in system memory.

 

[madshi]

 

[KimB]

I wonder if windowed mode screenshots are any different?

 

[Brent]

I have implemented a way to record your own flyby paths in my framework and a way to play them back. This way all my demos can be used as benchmarks. What's left to be done now is log framerates etc. So I wanted to ask in what format it's most desireable to have the output in and what information you want to be available there, fps, time/frame, etc?

Also, demos would have to be recompiled for this to work though, so what demos would you like to be recompiled with benchmarking support?

I am highly looking forward to this Humus

The format of the output, it can simply be a .txt file, or if you want to get fancy an .xls file that formats to a spreadsheet. But text files are simple enough really.

As for the Info in there, min/avg/max fps would be nice. How many frames rendered. I don't know if it would be possible to show the polygon/fill-rate, but that would be neat.

Maybe a way to show to fps over time, so a histogram could be created.

Thats about all I can think of right now.

 

[Reverend]

Humus, just make a one level playable "game" with the usual benchmarking mode.



Brent, how different is using Humus' technology demos different from those in 3DMark03, and how can you use Humus' benchmarkable demos in [H] video card reviews when [H] wants benchmarks using "actual game engines"?

Brent, sorry, couldn't resist. All in good humour, of course I just finished reading almost all 30+ pages of a thread over at R3D re the FM/NV fiasco and found some of Kyle's comments rather amusing.

 

[Brent]

Humus, just make a one level playable "game" with the usual benchmarking mode.



Brent, how different is using Humus' technology demos different from those in 3DMark03, and how can you use Humus' benchmarkable demos in [H] video card reviews when [H] wants benchmarks using "actual game engines"?

Brent, sorry, couldn't resist. All in good humour, of course I just finished reading almost all 30+ pages of a thread over at R3D re the FM/NV fiasco and found some of Kyle's comments rather amusing.

I love to look at all types of benchmarks, synthetic, game, whatever

If it looks usefull I'll consider using it

 

[Dave Baumann]

Rev, you're behind the times - I already asked that one.

 

[Reverend]

Dave, I don't have the time nor the concentration to browse forums other than our own

I must've missed it -- what was the answer to the same question you and I asked?

 

[Humus]

Alright, so it's done.
Information on my site:
http://esprit.campus.luth.se/~humus/

I log the time it took to render each frame. This should give you all information for any kind of statistics you want. Average/min/max/median/etc.

I've only updated VolumetricLightingII so far, will update more demos soon.

 

[LeStoffer]

Alright, so it's done.
Information on my site:
http://esprit.campus.luth.se/~humus/

I log the time it took to render each frame. This should give you all information for any kind of statistics you want. Average/min/max/median/etc.

I've only updated VolumetricLightingII so far, will update more demos soon.

Brilliant! 8)

Just one thing I have to ask you about: Since you have developed these nice demos on ATI hardware (8500 and later 9700 P AFAIR) there is bound to be some questions about whether they take optimal advantage of nVidias hardware - especially the FX line with their 'twitchy' shader architecture.

It may not mean a damn thing, but you might wanna consider doing some kind of disclamer anyway before the benchmarking takes of in a big way.