Use of Custom Demo's In Reviews

[Miksu]

Didn't first notice that those Nascar4-scores were a bit of strange too. In the older test Radeon's were leading the bag in most of the tests, now they're behind with like 20fps all the time. So we go from this http://firingsquad.gamers.com/hardware/msi_geforce_fx5900-td128_review/page6.asp to this http://firingsquad.gamers.com/hardware/evga_e-geforce_fx_5900_ultra_review/page6.asp

yes it's strange - did I read it correct, NASCAR is now more stressful, but SS2 and Q3A are less stressful?

Yep, if I read this correctly: "Like the Serious Sam demo we made, this new demo is less stressful than the demo we used earlier, as Q3DM7 is a much larger map than Q3DM1, hence the higher frame rates."

It's somewhat odd that when creating new timedemoes, they're choosing less stressful demoes to some games and more stressful to others. Then in UT2k3 they're using the original demoes.

edit: It seems that eVGA's board is giving many problems to users: http://evga.com/community/messagebo...ra+issues&Forum_Title=GeForce+FX+Family+Forum

 

[Sharkfood]

It's somewhat odd that when creating new timedemoes, they're choosing less stressful demoes to some games and more stressful to others.

I don't see how it's odd at all. Like I mentioned earlier, a benchmark reviewer can basically make any statement of performance they wish- just by using selective benchmarks, resolutions, settings and feature sets.

It's old hat and has been used for a long, long time. Once you have identified a particular strength or weakness to a few platforms, you just have to work these in.

The tough part is trying to quantify how much a particular condition may actually occur during your average game. This is what draws the line- if benchmarks are focused 100% on a particular performance element that only is encountered in like 5% of your average playing encounter.

 

[chavvdarrr]

On the topic - guys from ixbt.com/digit-life decided to change used programs/demos for benchmarking... They'll use (excerpt from http://forum.ixbt.com/0010/044212-44.html)

1. SSAM: TSE
only with new custom demo
2. RtCW
same
3. Q3
same
4. UT2003
same
5. Game2/3DMark2001
will be changed to Unreal II (therough 'BenchEmAll')
6. RightMark
7. AquaMark
will be changed to Unreal II (therough 'BenchEmAll')
8. Code Creatures
This won't be changed although there are "cheats". But as there are no better game-like shader-test will use it for now. It will be changed when we find something better.

Custom demos won't be published! After releasing anti-detector scripts there is high attention from NV/Ati engineers to our reviews so we decided not to publish used demos!

 

[kihon]

Has anyone actually tried comparing standard time demos and custom ones while running the antidetector scripts? Im curious to see if the standard timedemos are affected more by the scripts than custom ones.

 

[deviantchild]

If you wanted to totally annihiliate a PowerVR board- just launch a lot of rockets.

It wasnt too long ago I was playing games on a Kyro1 and I dont recall experiencing 1-2 fps at any time during Quake...
Though I might have been having too much fun to notice.....

I dont think Sharkfood is talking Kyro..

Ok, which PowerVR board was he talking about then?
Sorry if I'm showing blatant stupidity :?

i have experienced this with Kyro but not with GLQuake,
i think i only experienced it with "GLQuake Update" and possibly also the engine that Nehahra used too
iirc it seemed that the smoke trails from rockets and the grenade launcher used a different method than that in GLQuake [don't ask me, i'm not up on this stuff like most of you] and this killed Kyro framerates
i don't know what it's called but i think they were using what looked like crosshatched textures (alternating colour and transparent pixels) to simulate alpha for these effects