Use of Custom Demo's In Reviews

DaveBaumann said:
Why? They are more likely to bear some relation to a game you are likely to play than the prerecorded ones, since most of the time they are just made of games we've played.

Anyway, there is a slight variation on a theme...
Click to expand...

I agree with this site as credability speaks for itself, while some others I shall not name, it would be hard not question if the demo is truly 'secret'...you know somehow it slips into a email message by mistake :p
 
all you have to do is do it 3dgameman style.

Instead of just a chart with the numbers for the Custom demo. make videos of the demo actually being run, and then a screenshot of the monitor with the results shown in detail. So people could see the final scores being posted on the monitor.

Lots of bandwidth needed for that though ;)
 
Brandon has posted the Nascar screenshots. He was forced to take actual shots with a camera.

Although the AA looks better, it's still a blurred mess...
 
McElvis said:
Brandon has posted the Nascar screenshots. He was forced to take actual shots with a camera.

Although the AA looks better, it's still a blurred mess...
Click to expand...

Yeah the blur is pretty appearent from what I can see it looks like Quincunx blur filter.
 
So what will it come down to? Will a reviewer have to become intimitely knowledgable about each demo, knowing its quirks on each card (per Sharkfood's Morrowind post), and thus create multiple timedemos to reveal overall performance, with graphs being normalized scores encompassing multiple timedemos, and a table below with individual timedemo scores for reference? That sounds like way too much work.

On the plus side, perhaps it'll wean websites away from posting rushed and usually incomplete (or limited) reviews the day of a card's "release." They'll be forced to take their time and do it right.
 
Uttar said:
It worked in 1998. In 1999. In 2000. In 2001. In 2002. Maybe they didn't do it to such a level. I don't know. But they did cheat according to some people here and at other places.
So, well... "Why not in 2003?"...

I've got only one explanation:
We're becoming smarter 8) ;)


Uttar
Click to expand...

NO! We are more observant.

I always found it strange that my Radeon's 2xAF done a better job than NV's 4xAF. I also found myself only using Omega's drivers for my former NV20 as they did a damn good job of increasing IQ by removing all the damn blur which occured regardless if AF was on/off.
 
Pete said:
So what will it come down to? Will a reviewer have to become intimitely knowledgable about each demo, knowing its quirks on each card
Click to expand...
And this is exactly what we saw happen with the Splinter Cell anomolies. We have good reviewers who know the benchmarks. I simply cant go back to places like Tom's anymore and read a review, without going WTF?
 
A website with a couple of new demos every few weeks for each of the popular benchmark games (Splinter Cell, UT2K3, Quake3, RtCW, SS:SE, etc, etc) would work I think. Reviewers and readers can download the demos from that one central site if they want to compare performance, but only demos that are newer than the drivers used in the review should be allowed.
 
Uttar said:
It worked in 1998. In 1999. In 2000. In 2001. In 2002. Maybe they didn't do it to such a level. I don't know. But they did cheat according to some people here and at other places.
So, well... "Why not in 2003?"...

I've got only one explanation:
We're becoming smarter 8) ;)
Click to expand...
No, no...I don't think that's it. I think it's something involving mirrors or Bush or something like that... ;)

Brent said:
Doomtrooper et al, we do know like everyone else about all the timedemo issues.

In fact Kyle has already asked me what I think about recording our own timedemos and not releasing them to anyone.

Please don't think we are oblivious to what is happening out there, we do stay informed.
Click to expand...
That's my big problem with you folks over there right now, you CAN'T be oblivious to it...yet you don't SAY anything about it! o_O

I thought part of your site's supposed purpose was also to INFORM your members about what was going on...not just you folks yourselves knowing it! :devilish:
 
We've all pretty much agree'd to lay off [H](Brent specifically) in another thread. He's a valuable part of these forums, and I would hate to see him stop coming here ;)
 
So what will it come down to? Will a reviewer have to become intimitely knowledgable about each demo, knowing its quirks on each card (per Sharkfood's Morrowind post), and thus create multiple timedemos to reveal overall performance, with graphs being normalized scores encompassing multiple timedemos, and a table below with individual timedemo scores for reference? That sounds like way too much work.
Click to expand...

And why not? They did all that work in the past, but for nothing so noble as to be "objective"- but quite the opposite. :)

The Quake1 + PowerVR example was a good one. If you wanted to totally annihiliate a PowerVR board- just launch a lot of rockets. Voodoo boards at the time chugged through them pretty okay. PowerVR boards choked down to 1-2 fps. If you wanted to sell 3dfx hardware, you just featured benchmarks that had 5-6 of your best friends on a rocket firing fest. The graphs made the PVR hardware look like it was beaten 15-fold.
 
Sharkfood said:
If you wanted to totally annihiliate a PowerVR board- just launch a lot of rockets.
Click to expand...
It wasnt too long ago I was playing games on a Kyro1 and I dont recall experiencing 1-2 fps at any time during Quake...
Though I might have been having too much fun to notice.....
 
Doomtrooper said:
I told Kyle and Brent their desire to use time demos from games will just shift the attention to 'optimizations' on these timedemos, anything with shipping time demo on a CD like Serious Sam 2 and UT 2003 are targets, even [H]'s own UT 2003 timedemo as it is freely downloadable.

I've watched Nvidias scores increase after driver release on their time demo...

42.72 driver [H]'s UT 2003 Benchmark 350/350

1047311589S4MzihRI0V_6_3.gif


44.03 driver [H]'s UT 2003 Benchmark 400/350

1053587960qo2f5WxATf_4_3.gif


5600 Ultra scores 13.4 with the 42.72 driver (the day the benchmark was released to the public) here:

http://www.hardocp.com/article.html?art=NDQ0LDY=

Then a 6 weeks later the magical 44.03 drivers increase [H]'s benchmark in UT 2003 to 39.7...now a 50 mhz core clock doesn't triple performance :LOL:

Every website should be using their own custom timedemo and NOT allowing it to be downloaded, and I suggest contacting some clans on the ladder in UT 2003 as I've played against them/with them in my day and would be a far more accurate real world stress tests :)
Click to expand...

You're comparing two different testing situations there. One uses the 42.72 driver, 2x FSAA, and 8x AF, the other uses the 43.03 driver, 2x FSAA, and no AF.

It is a known fact that nvidia increased AF scores dramatically in the 43.51+ drivers. What exactly they did to increase scores remains a mystery. But the fact remains that when using drivers prior to 43.51 enabling AF incurred a larger performance hit. Couple this with the increase in clock speed, and there is nowhere near a 3x performance increase. Although nvidia might have done some "optimizing" for the [H] UT2003 benchmark, your post does not include enough evidence to draw such a conclusion.
 
What about a different reveiw for each new set of drivers?

This way with a given set of drivers you can run your new top secert demo, post results on your killer review then give linkage to the masses. This allows for some level of comparison to the public.

But this also helps to prevent demo level optimization as when the IHV says here are our new drivers you, smile and say thanks, then use a different demo to test. This way the IHV never really knows all of the demo details unitl its made public.

However this puts more work on the review sites.

I would also recomd maybe some of the bigger sites working together. To share these demos to help keep some consistancy.

Or was this what Sharkfood was getting at? If so my bad.
 
StealthHawk said:
You're comparing two different testing situations there. One uses the 42.72 driver, 2x FSAA, and 8x AF, the other uses the 43.03 driver, 2x FSAA, and no AF.
Click to expand...

Yep I linked the wrong graph, still a significant increase without AF..so my assumption is just fine.

1053587960qo2f5WxATf_4_4.gif


1047311589S4MzihRI0V_6_3.gif


Still over double from the 42.72 drivers:

2X FSAA/8X AF 42.72: 20.2
2X FSAA/8X AF 43.03: 51 FPS !! or almost 2.5 times the performance on the same benchmark, and no 50 mhz core clock doesn't account for it.
I compared no AF scores also the speed improvement is the same, so your optimized AF idea is wrong.
 
One of those graphs is the older type, while one of them is of the newer type. Leading me to think there was some time between the reviews. Are both review PC's the same?
 
micron said:
Sharkfood said:
If you wanted to totally annihiliate a PowerVR board- just launch a lot of rockets.
Click to expand...
It wasnt too long ago I was playing games on a Kyro1 and I dont recall experiencing 1-2 fps at any time during Quake...
Though I might have been having too much fun to notice.....
Click to expand...

I dont think Sharkfood is talking Kyro..
 
Randell said:
micron said:
Sharkfood said:
If you wanted to totally annihiliate a PowerVR board- just launch a lot of rockets.
Click to expand...
It wasnt too long ago I was playing games on a Kyro1 and I dont recall experiencing 1-2 fps at any time during Quake...
Though I might have been having too much fun to notice.....
Click to expand...

I dont think Sharkfood is talking Kyro..
Click to expand...
Ok, which PowerVR board was he talking about then?
Sorry if I'm showing blatant stupidity :?
 
You must log in or register to reply here.
Back
Top