Futuremark's First Monthly Media Newsletter Is Here!

[aZZa]

I cant exactly back Kyle but everyone deserves their OWN opinion and say in this world. Futuremark has unfortunately been slapped an unreliable tag on its 3DMark03 version, which is probably one of the best futuristic 3D video card benches in recent years. Shadermark proves the shading performance (especially in earlier nVidia drivers) of the Radeon series, however without the NEW DX9 series/equivalent OGL games/benches it seems difficult to the less educated that some things may actually show more than they seem to show....

3dMark04 on the horizon near???

Anyway good luck to all in the mobile phone/pda series of 3D benchmarks coming soon... any powerVR here... finally MBX coming soon to upstage the big guns of 3D on the small stage

 

[digitalwanderer]

1. No demo's on the rails... take levels using real game engines and allow the user to play through them whilst 04 records performance. You could have synthetic versions which are on rails but are their for info only/dont contribute to the final score.

That would defeat the purpose of the benchmark for making an accurate comparison 'tween rigs, it adds too much of a random factor into it.

If your second idea is well implemented, (which they ARE working on ), then it shouldn't be a problem.

And to Azza: Yes everyone is entitled to their own opinion, it's just in this case Kyle's opinion has been repeatedly proven wrong and he still maintains it in spite of the evidence...it's obvious he's got a woody against Futuremark right now although I won't speculate about why. My big question is why would you want to try and let an illogical and irrational person add input to your project when they've been bending over backwards for well over a year to prove their bias against it?

 

[Bouncing Zabaglione Bros.]

Well except for that being a genuine crazy thought and a terrible plan it's...well, it's crazy.

Kyle has just done too much and held his wrong views too hard in the face of all the evidence...what's next, make DH a member since they crapped on it too? :|

Why is it crazy to have someone who has oposing viewpoints in on the product development? It means that the final product is more balanced. Additionally it would also benefit Futuremark to have one of the (if not the) biggest tech websites onside.

It's only useful when the person with the opposing viewpoint has it based on information and facts, rather than just a stubbornly held opinion because his friends at Nvidia "said so".

Really, I don't think Kyle has either the technical knowledge or experience to be useful to the FM program. It would be like asking an icehocky fan to be on a football commitee just to ask him about his differing viewpoint - it's just not going to tell you anything useful.

 

[Veridian3]

That would defeat the purpose of the benchmark for making an accurate comparison 'tween rigs, it adds too much of a random factor into it.

Depends on whats more important from an end user POV, an accurate synthetic performance result or a comparable real world performance comparison. Maybe its time for FM to shift emphasis from one to the other?

Actually, if you went with both of my options, one on rails and one playable you could give 2 scores... Synthetic and Real world. Let the users decide which to quote/discuss and publish.

 

[digitalwanderer]

Depends on whats more important from an end user POV, an accurate synthetic performance result or a comparable real world performance comparison. Maybe its time for FM to shift emphasis from one to the other?

I disagree. If you want to get an accurate reflection of how a card will respond in a "real-world gaming situation" I highly suggest you try a few real-world games and bench 'em, synthetic benchmarks serve a very specific purpose and I kind of like 'em the way they are.

 

[Bouncing Zabaglione Bros.]

I cant exactly back Kyle but everyone deserves their OWN opinion and say in this world.

I have the opinion that the world is flat and the moon is made of green cheese. Does that make my opinion valid, factually correct, or deserving of serious attention just because it's "my opinion"? Of course not.

I wish people would drop this "everyone's entitled to an opinion" thing. Yes they entitled are when it is something subjective like "is strawberry or chocolate ice-cream better?". When it is something objective such as "Do Nvidia have worse AA quality than ATI?" or "Does Nvidia reduce Trilinear filtering to Bilinear whilst lying about it?" there are no "opinions". There are only facts. There's the truth, and then there's everything else.

Your "opinion" is either based on correct facts or incorrect facts. Kyle has demonstrated the latter time and time, and is why he gets ridiculed and derided on a regular basis.

 

[aZZa]

V3 .... I'd love to be able to display the theoretical capabilities ....alongside the real-world abilities in the benchmarks, but as has been shown in recent times, there is always ways around doing/displaying the right results.

Even adding a random (and repeatable) seed/view will not stop the damaging driver quirks applied to common benchmarks... unfortunately unless patches maybe released (to prevent unruly behaviour) or final benchmark programmes developed outside graphic companies reaches in order to stop the optimisations. I cant see there being a way to prevent a programmer finding a way to beat any detection mechanism to this in the long run??? 3DMark 2001 was riddled with many questionable optimisations, why did it not receive the same scrutiny that 3DM03 received? Its funny how a gf4 can drop off fps from the 40.x drivers to 56.x with this wonderful new compiler and driver within the nature01 pixel shader bench if it is fact doin its prescribed job? It doesnt make sense???

In reality as long as the programs work... it doesnt make a difference, but it would be nice if one could have the option to run a game/bench/program in an optimised/normal/self-descibed way (profiles...??) without being forced to play it the way the graphic company wants it run so they look good for the reviews against rival companies. Lowering quality to increase fps should be frowned upon by reviewers, as for eg 120 vs 100 fps means nothing especially if 100fps looks 200% better than 120 fps, and should play exactly the same but look that much better.

Also... Good luck to Futuremark in approving drivers to prevent the questionable optimisations from ruining the stats in their benches from being published on the web.

 

[aZZa]

Bouncing Zabaglione Bros... Im not saying anyone is right or wrong. I think u can understand where i come from. We have idiots in this world blowing up planes and trains to prove a point they are too gutless and scared to reasonably speak out about. This is a rather trivial issue.

Kyle, right or wrong deserves his say. He, however does not have the right to slag off someone/something ( ie Futuremark) without any reason. This I guess is why he has found himself in trouble with a court case of his own with the phantom console group.

Futuremark has unfortunately made a few blunders in its enforcement of its benchmarks over the years which has led to people questioning their current authenticity. They do serve there purpose if used right and Kyle might be and is probably wrong about this, but until some more benchmarks/games come out to prove what has been presented in 3DMark03 is likely to occur in future programmes he will continue with his own opinion like many people in the world.

The unfortunate situation is that the facts have been slurred/hidden and been undetected for too long for this situation which has been going on for a lot longer than many people realise. Most companies have been at fault at some time or another in this quest for the best public performing hardware. Some people have been influenced to believe some of these benchmarks on all there merits without the thorough investigations.

I hope for all, S3, PVR, XGI and co can release bring some quality to the benchmark playing fields to keep up the standards up of the big guns, and force some competition which has been lacking in recent times.

 

[digitalwanderer]

Kyle, right or wrong deserves his say. He, however does not have the right to slag off someone/something ( ie Futuremark) without any reason. This I guess is why he has found himself in trouble with a court case of his own with the phantom console group.

Uhm, no. Kyle is actually right on that one and what he said and the way he's defending himself is a genuine GOOD THING(tm) and is deserving of some respect and admiration of which I gladly give him for his stance and the way he's handled himself with the whole Phantom story/lawsuit/dealy-thing.

His stance on FM and 3dm2k3 however is not, and his reasons and oppositions to it make sense only to Kyle in whatever whacky world Kyle lives in.

There's actually a big difference 'tween the two incidents and attitudes, I just thought it worth mentioning. Kyle isn't always wrong and he ain't always an idiot, but his stance on 3dm2k3 sure don't make any sense unless we go into some weird conspiracy theories which could technically get me in trouble.

 

[Nazgul]

Depends on whats more important from an end user POV, an accurate synthetic performance result or a comparable real world performance comparison. Maybe its time for FM to shift emphasis from one to the other?

The problem with your "real world" description it that it's a non-repeatable test. In order for a benchmark like 3DMark to be useful, it has to be repeatable. If the image being displayed onscreen during the test run is based on the actions of the user, then it can't be precisely repeated. At most, you can get "pretty close", possibly even "close enough."

Plus, the idea that an "on-rail" test is not "real world" is bunk. It's no less "real world" then running one of the built-in timedemos that comes with an actual game. It's just not using the rendering engine of an actual game. And that in no way makes it invalid. It's still doing Direct3D rendering of a full scene with a variety of geometry, effects, shaders, etc. to create images and movement comparable to what is seen in an actual game.

What -would- be interesting is if 3DMark could include the ability for the user to create their own custom route/walkthrough/flyby/whatever of a scene that they could save and then use for a series of tests on various machines. I have no idea if this could be done, and what sorta issues might have with it, but it would seem to combine the best of both worlds.

 

[Tim]

Kyle, right or wrong deserves his say

He has had his say and been proven wrong. When someone is proven wrong and keep repeating the same nonsense they deserve to ridiculed.

 

[digitalwanderer]

What -would- be interesting is if 3DMark could include the ability for the user to create their own custom route/walkthrough/flyby/whatever of a scene that they could save and then use for a series of tests on various machines. I have no idea if this could be done, and what sorta issues might have with it, but it would seem to combine the best of both worlds.

OOoooooooh! I LIKE that idea, I just wish I would have been the one to have suggested it!

It's a great solution to make it both repeatable and non-cheatable. 8)

 

[kyleb]

non-cheatable is streaching it, less cheatable though.

 

[AzBat]

Worm and Future -crew,

I like the newsletter. Good job on it. I think that should help things in the online community, but I don't think it will help greatly with 3DMark03. Should do wonders for the next offering. Keep up the good work.


As for the suggestion of bringing Kyle into the program, I think Futuremark should extend the offer to all the major 3D review sites out there. Like they did with the Benchmark Bootcamp event held years ago in San Francisco. BTW, Kyle and company was there. It happened to be the first time I ever met him. Anyway, I believe having HardOCP involved would be a good thing, not just Kyle. Though I believe Brent would be a much better liason between HardOCP and Futuremark since he does the 3D reviews and I think he understands Futuremark better.

My 2 cents.

Tommy McClain

 

[digitalwanderer]

non-cheatable is streaching it, less cheatable though.

Hush you! Let me enjoy my fantasy world a bit longer...

 

[Pete]

]Oh well, at least some room for improvement for the next issue!

That's the secret to a successful business, no?

If you are headed for GDC, do go and meet up our chaps. At least Tero Sarkkinen & Patric Ojala will be there.

Ha! Given the profound ignorance I displayed WRT to NV2A's vertex shaders, I don't think I'm the type who could benefit from GDC. Rest assured that if I were going, though, I'd be interested in meeting the Futuremark crew.

 

[Hanners]

1. No demo's on the rails... take levels using real game engines and allow the user to play through them whilst 04 records performance. You could have synthetic versions which are on rails but are their for info only/dont contribute to the final score.

Okay... So what real-world game engine should FutureMark make use for the 3.0 shader tests?

I really don't see the point of a packaged benchmark that tries to be exactly like a real-world game, right down to the engine used. If you want that route then why not.... Oooh, I dunno.... Just benchmark using real-world games?

 

[Tim]

Okay... So what real-world game engine should FutureMark make use for the 3.0 shader tests?

Yes a real world game engine in 3dmark would not make sense at all. 3dmark needs to be progressive in its use of features, mimicking current games would make it almost completely useless.

 

[Veridian3]

One of the criticisms of 03 was that it was too theoretical and didnt show in the best possible way how your system would perform in games. Using real game engines would resolve this issue.

You could bench with real world games, but if you had maybe 4 different game engines in 04 it means everyone can run all four on their system and compare performance. Not everyone has access to loads of games to test their system and see how it compares to the many reviews they see. Including real game engines in 04 doesnt stop reviewers from being able to test other games too...and it doesnt stop futuremark from including theoretical tests.

It also wouldnt be hard to arrange engines, i'm sure quite a few companies would be falling over themselves to get their engine featured in 04. Its a huge advertisment for them.

As for Pixel Shader 3.0, who says there will be a test in 04 that counts towards the score and features PS3.0...it would seem more likely to me that there will be a theoretical test featuring 3.0 ... much like the PS1.4 test in 3dm2001. Depends on when its released, and what ATI and NV have released i guess. There is also the option to have a company who plans to use PS3.0 in an engine write a test for FM.

 

[Quitch]

Using a game engine would be pointless, why not simply use a real game? How would 3DMark be any different from any game benchmark you could run if it used a game engine?

The very fact that it doesn't use one is what makes the thing useful!