[H] Benchmarking Future Ed.

[Mariner]

Very good posts from BrainPimp over on that forum. It looks as though he must have been a poster there for quite some time and he doesn't appear to have any axe to grind.

I'll keep an eye on that particular thread to see if Kyle does respond to any of his points.

The problem is that Kyle is not just "defending the indefensible", he's "not even mentioning the indefensible!" - especially on his own web site.

 

[LeStoffer]

I’ve had a number of phone conversations with NVIDIA since this 3DMark issue started and their point of view is that demo’s and benchmarks such as 3DMark and Shadermark are not representative of games because they weren’t coded specifically with NVIDIA shaders in mind, but games on the other hand will be – by either the developer doing it off their own back or games that go through the TWMTBP campaign. This is the issue that is key to the approach to ‘Application Specific optimisation’ in benchmarks they appear to feel they have the right to change the shaders from those that were coded because this will actually reflect game code more.

Thanks for the insight. This tells a lot about the level of arrogance that their marketleader status have brought them.

 

[WaltC]

The issue I have wrt Kyle specifically has to do with the fact that his opinion of Beyond3D is based on non-Beyond3D staff (i.e. open foum participants') expression of opinion. He is saying, basically, that "I have no respect for Beyond3D because their forum particpants are flaming me".

If he could address what Beyond3D -- the site, its staff, its partners -- are doing wrong, I wouldn't have a problem with him. But he is treating B3D's forum participants' opinions as officially representative of Beyond3D. That is wrong.

I have yet to see Kyle quote a substantive comment made in open B3d forums and respond or engage in dialogue on the substantive points made in the open forums wrt his opinions. I'd say the ratio of substantive remarks to non-substantive emotional expressions is on the order of 10-1, at least. Kyle is not only mischaracterizing the official B3d positions, he is also mischaracterizing open forum opinions as well.

 

[WaltC]

I've had that in my Sig since before all this stuff started

I'm a fan of Buffy the Vampire Slayer, and thats the best thing I could think of that fit that tag line, even though it really doesn't make much sense, heh

it doesn't relate to any of this

http://www.hardforum.com/showthread.php?s=&postid=1024948932#post1024948932

Brent, thanks for shedding some light on that for me... I guess I never noticed it until now. You have to admit, though, that under the circumstances this is a sig that will be easily misunderstood, and so I'd think you'd want to change it. I mean, whether they are timedemo benches or synthetic benches or whatever, I'll wager your reviews will contain benches of some sort--so you might want to consider the tag from a fresh perspective based on the opinions as served up by [H.]

Why not "the Benchmark Cheater Slayer," instead....

 

[nelg]

Has there been a situation like this before. Where with a pervious generation of hardware was there a time when two competing products performed similar to on another but some synthetic test showed that one of the cards may be better suited for future games ? If so what was the outcome ? Were the synthetic test correct in their prediction that one of them would be better suited for future games?

 

[WaltC]

...
I agree with Kyle's assessment of this forum as overly vicious. Sure, we may know more than other forums, but who gives a flying Dutchman when all you read is how someone was paid off or spewing garbage, etc. A little more civility would suit this forum well. I'd rather have Kyle in here to debate with, even if I disagree with him. Chasing him out with rather rude remarks (like he was so easily able to find) isn't really doing anyone a service.

According to the testimony I've read from several people who post here, Kyle is no stranger to such remarks in the [H] forums. The difference seems to be (according to these people) that Kyle is less threatened in his own forums because he can delete posts and ban members who do not agree with him. He can not do so here, and that seems to be why he doesn't post anymore--if you want to overlook the very germane fact that his opinions on these matters would be exceedingly difficult to justify.

Also, I think you are viewing some of the more "colorful" comments here out of the context of Kyle's "colorful" editorials as presented at [H.] I doubt I will ever forget his "Two days after the Doom 3 preview..." slander of the ET site. Then there is his official "policy" on FutureMark's software, which coincidentally seemed to gel only after nVidia withdrew from the 3DMark program and made negative comments about the software. Then you have the "If you can't see the cheating it's not cheating remarks," which applied to web sites who could not only see the cheating, but published *screen shots* of the cheating (which apparently Kyle was unable to see.) The list in this sad story just goes on and on, but overall I find the occasional emotional remarks people make not to be vicious at all, but actually quite understandable given Kyle's reluctance to engage in a dialogue here about his editorial opinion. Most people have little tolerance for being told that black is white and that up is down, and that kind of thing. Most people have little tolerance for being told that benchmark software is bad, but that companies who cheat the benchmarks are blameless. I know I do... Certainly. It's insulting.

 

[Brent]

I've had that in my Sig since before all this stuff started

I'm a fan of Buffy the Vampire Slayer, and thats the best thing I could think of that fit that tag line, even though it really doesn't make much sense, heh

it doesn't relate to any of this

http://www.hardforum.com/showthread.php?s=&postid=1024948932#post1024948932

Brent, thanks for shedding some light on that for me... I guess I never noticed it until now. You have to admit, though, that under the circumstances this is a sig that will be easily misunderstood, and so I'd think you'd want to change it. I mean, whether they are timedemo benches or synthetic benches or whatever, I'll wager your reviews will contain benches of some sort--so you might want to consider the tag from a fresh perspective based on the opinions as served up by [H.]

Why not "the Benchmark Cheater Slayer," instead....

 

[WaltC]

Has there been a situation like this before. Where with a pervious generation of hardware was there a time when two competing products performed similar to on another but some synthetic test showed that one of the cards may be better suited for future games ? If so what was the outcome ? Were the synthetic test correct in their prediction that one of them would be better suited for future games?

I can't think of a single instance in which an IHV in the past has said, "Don't believe the benchmarks, they're all wrong." It's tantamount to saying, "Don't believe what you can see, believe instead what we tell you." That is a tactic guaranteed to fail and to backfire on the company employing it.

I can think of instances where IHV's complained about this or that aspect of a particular benchmark, but that's a far cry from telling people to discard them in toto, as we're seeing now. When 3DMk01 routinely spit out scores showing nVidia products ahead, did nVidia withdraw from the program and seek to discredit the benchmark at that time? Obviously not.
Did [H] publish editorials declaring that "the frame-rate benchmark is dead"? Of course not.

So what's changed? All that I can see that's changed is that nVidia is no longer ahead, and nVidia doesn't much care to be in that position, so the company is enlisting whomsoever will to aid it in a vain attempt to stamp out all software that doesn't portray nVidia's hardware as nVidia prefers. This is the new wrinkle to the 3D competitive story as I've seen it unfold over the past several years. When ATi was behind 3dfx and nVidia a few years ago did ATi suggest that benchmarks were wrong and that people should therefore abandon them? Heh--they'd have been laughed to scorn had they done so. Indeed, nVidia itself had nothing whatever against frame-rate benchmarks whenever it could use them to one-up 3dfx back then. The difference is that ATi endured the benchmarks until it could produce products which ran better than the competition's. nVidia's attitude is much different--they want to convince people to throw out any software which demonstrates their products are second rate compared to the competition's. Heh--nVidia has struck the motherlode--of pyrite... nVidia doesn't need luck so I won't wish them that--it needs better products, instead. If and when that happens for nVidia, all of this anti-benchmark talk will fade into nothingness...

 

[Hanners]

Why not "the Benchmark Cheater Slayer," instead....

[nVidia lawyer] Excuse me Sir, that should say 'the Benchmark Valid Optimisation Slayer'. Thank you for your co-operation. [/nVidia lawyer]

 

[Reverend]

If he could address what Beyond3D -- the site, its staff, its partners -- are doing wrong, I wouldn't have a problem with him. But he is treating B3D's forum participants' opinions as officially representative of Beyond3D. That is wrong.

Heh, I think he did that here Rev:

Anthony Tan came to me a couple of months ago and asked for the lead VidCard reviewer’s job here at HardOCP. He was turned down. I think that has lead to some of his recent animosity towards us.

First it was Extremetech that was pissed cause they didn't get to bench doom3 like he did. Now you disagree with him because you didn't get the Lead VidCard reviewers job. The guys got an ego a mile wide. And you wonder why he gets dumped on here Russ?

Kyle's reasoning is wrong -- I have no animosity towards him for the reason he gave. Shit, I didn't want to bring this up but since Kyle saw fit to bring this up, here goes :

Yes, I did ask to write reviews for [H] but I wanted the Top Job (so to speak), he said no because he didn't want to piss off Sean and Brent (bu letting me/someone "leap frog" them) and he thinks they're both doing a good job. Actually, when I wrote him in my email asking for such a position, I clearly stated that I understand this may be difficult for him to do because I know both Sean and Brent has been doing good stuff for his site. So, anyway... he turned me down and I wrote the following :

Like I said in my original email, I understand the situation and the difficulty you may face.

He'd also asked to talk to me on the phone about this but I said I'm in Malaysia and that it may be too expensive for either of us. It ended there and then.

It is worth noting that back when I first left B3D (circa March 2002) and told some folks (which included Kyle) about this (because I didn't want any problems upon leaving B3D), Kyle immediately wrote me asking me if I would consider writing tech articles (not reviews) for [H]. I declined, becasue I was trying to set things up at Voodooextreme.

Anyway... Kyle is wrong if he thinks I have any "animosity" towards him. Since he brought this rather private correspondence subject up in public, then I would now feel no hesitance in quoting here what I wrote him a little while back - I won't quote ad-verbatim his email replies to me though as I will respect his wish that I don't post his emails in public. Here's the entire correspondence on a subject I wrote him about, which was titled "Reviewing video cards according to you" in early June (which was prompted by his almost-condemnation of Futuremark/3DMark03) :

Bro,

I agree with your assertion that using games as benchmarks for evaluating video cards is important.

But by doing so, you are probably also saying, in a hidden way, that the latest features and technologies in the newest video cards should be ignored, or at the very least, taken in with only passing notice. Why? Because there will never be any games available that will show off the latest features/tech in the latest video cards at the time of such latest-and-greatest video card review.

So, while I agree with your stance/opinion that using games is the most important basis of a video card review, I hope I never see any of your future video card reviews containing any mention of the latest features and technologies of that video card. Example, in case you don't understand :

When we get DX9-extended (i.e. VS and PS 3.0) video cards, in your review of such video cards, you should NOT mention at all about the DX9-extended features/technologies of such video cards, because it will serve absolutely no purpose since you will be using existing games, which will never feature DX9-extended features. You should focus, and devote the entire content of the review, on the performance of the card using available games/game demos.

Thanks for reading, hope you'll have a good weekend.

His reply (paraphrased of course) was that (he actually started by saying thanks for my thoughts) he thinks I'm one that listens to what people say he says, and thathe has never stated synthetic benchmarks should not be used. He said that I'm simply being an ass and that it shows. That I should not waste his time with such "drivel" wher he says should better belong to flaming and personal attacks in the forums I attend. He said he has lost all respect for me.

I replied :

Well then, everyone shows he's an ass, at one time or another.

Actually, I haven't been reading too many forums about what people think of you or what they say you say. In fact, the only forums I've been reading are my own for the longest time that I can remember (and just for today, I started at 3DGPU).

Anyway, what are your thoughts on 3DMark03's pure feature tests? Forget the four Game Tests -- what about its feature tests? I'm a little confused -- you've been saying bad things about 3DMark03 as a whole. But 3DMark03 contains *very* relevant and important synthetic feature tests.

Here's a link for you where I express my honest thoughts about what and how I think of 3DMark03 :

http://www.beyond3d.com/forum/viewtopic.php?t=6225&start=20

Please read my two posts on that page, and pay special attention to my second one. I don't think 3DMark03 is a "Gamers Benchmark". I don't think it, or future 3DMarkXX iterations, will ever correctly approximate what games in the future will be coded or even look like. I take 3DMark03 for what it's worth *in and of itself, nothing more*. But by doing so, I can gain useful information about the performance of current hardware vis-a-vis the very features it advertises, which we will never see in games this year or perhaps the next, and certainly NEVER looking like 3DMark03's various Game Tests nor coded the way of the Game Tests in 3DMark03.

I know where you stand Kyle. When I ran my own personal "3DPulpit" site, I never used 3DMarkXX -- in fact, I made it a point to mention that I don't use 3DMarkXX. The site of mine was focussed on using the latest gee-whiz video cards running available games. Hell, I never even tested forward-looking synthetic tests.

Now, as I said I don't visit forums other than my own, so I don't really know what you say in those forums. I only read what you say on your own site's front page and your articles. Such news posts and articles by you indicate that you are entirely writing off 3DMark03. I cannot understand why, because you have just said that synthetic tests can be used (in your email below) and 3DMark03's feature tests are as good a synthetic test as you'll find anywhere.

I have no stocks in Futuremark, nor do I particularly care if 3DMark03 is a succesful and popular application or not -- I just use some of its tests suites because I find them correctly indicating advanced features usage and performance.

Personally, I think you are simply just refusing to back down from your condemnation of 3DMark03 entirely because, well, you've gone too far down this road.

I hope you appreciate my frankness. I found your "Benchmarking For Gamers" pdf an extremely commendable endeavour.

Perhaps we both don't quite understand each other as well as we like. For that, I have to apologize if my below email is negatively presumptious. I mean well, and I know you mean well too. It's probably just that we seem to have very different takes on what the 3D industry is, or will be, and as we go along expressing our opinions on our sites or in various forums, we seem to be overlooking a variety of valid opinions in this matter.

I have learned that while I always seem to be convinced that I am always correct, I am also never far away from being wrong. It's a fine line, and it takes acceptance of opinions (while being calm!) for that fine line to go one way or the other.

Completely writing off 3DMark03 is not right because it has many useful features. The advertising of 3DMark03 by Futuremark is incorrect in my books but that does not dilute the fact that it is a useful gauge of things to come in certain aspects.

Sorry if you think I'm an ass. If I can't write you personally, and honestly, like I did below to express my thoughts on matters I think is better left in private correspondences, I will STFU rather than post in forums. Your "Please do not waste my time with such drivel again as it is much better suited for the flaming and personal attacks in the forums you attend." is out of line since I did not attack you in my email. I was just curious about your view on this matter. I do not ask for your respect, just your thoughts about what you want your site to offer to the public.

... to which he never replied.

I am not happy that Kyle chosed to publicly mention I approached him for a "job". More than anything else, this certainly should remain private.

 

[WaltC]

_______________
the Benchmark Cheater Slayer

...

 

[K.I.L.E.R]

More than anything else, this certainly should remain private.

It was a blatent attack, which he will obviously deny, nonetheless it is an attack.

I written up a short, and concise article myself stating "Attacking the Beyond3D staff is also low as they have done nothing to him.".

 

[Doomtrooper]

Open forums are meant to discuss peoples opinions, not everyone has the same opinion but that is what makes the world go around.
My problem is and has always been that he feels that once you make a decision, even if it is wrong (and this case is very wrong) if you stick to your guns it shows integrity. That couldn't be farther from the truth, admitting you were wrong is the 1st step in getting integrity back.

It doesn't take a rocket scientist to see what has influenced his opinion, and it is his little trips down to Nvidia HQ that did it.
It is also about qualificatons to be doing such editorials, such as 3D programming or working in the chipset industry, then you have knowledge to wade through the PR to see the truth.
This site does that as the staff and members have that knowledge.
I've have debated with him for years on forums, his opinions mean little to me as 'integrity and Kyle' can't be said in the same sentence.
Taking a application from a competing IHV and using it to get hits and try and corrupt another companies premier card launch, then totally ignore the FiringSquad findings in time demos, 3Dmark, IQ issues.
Journalists report journalism, and [H] only posts what it wants to, not what it should post.
He certainly shouldn't be as popular as he is, as a review site the only thing that makes the site somewhat worthwhile is Brent, and even that is not worth giving a hit from this PC and many aquaintances I have.

 

[Bolloxoid]

I agree with Kyle's assessment of this forum as overly vicious. Sure, we may know more than other forums, but who gives a flying Dutchman when all you read is how someone was paid off or spewing garbage, etc. A little more civility would suit this forum well. I'd rather have Kyle in here to debate with, even if I disagree with him. Chasing him out with rather rude remarks (like he was so easily able to find) isn't really doing anyone a service.

This viciousness is fully warranted. Kyle is an ass and he is suffering the normal social consequences for that. He has shown time and time and time again that he will always resort to completely inane and undue attacks and while bashing him might not be the best thing this forum has offered I think it is actually a good thing he is not posting here. You have much too great faith in his ability to "debate" things.

 

[Dave Baumann]

A little more civility would suit this forum well.

The level of vitriol is fairly irksome. I'd certainly hope that we could maintian a higher degreee of converstation about the points at hand here without resorting to the less constructive comments (Martox's last reply springs to mind as one example).

Come on guys, you know we operate a fairly open house, but that doesn't mean that you should abuse it. Lets try and be constructive in our critisism please.

 

[Dave Baumann]

Kyle is an *ss....

Sigh. Just as I was posting.

Again, these type of plain attacks are not welcome here - present you grievances more constructively please.

 

[Doomtrooper]

Dave I think the issue is this has been self inflicted upon Kyle, email exchanges between members and him including myself are always rude.
I was a member on the their forums, but quickly realized debating with Kyle there was useless as if you posted a reply he couldn't answer the thread was locked or deleted.

I wouldn't worry too much about the site image from this, mature people realize who is in the wrong here, and the forum members participants don't reflect the site.

 

[Dave Baumann]

Dave I think the issue is this has been self inflicted upon Kyle, email exchanges between members and him including myself are always rude.

Whether or not you or other feel he has brought this on himself is beside the point, you are being rude to B3D by bringing these types of attacks into this forum.

True enough, these newer forums are here to bring some of the less technical disccusion that some of our newer members wish to participate in out of the tech forums, but that doesn't mean they should devolve into flaming. I've got no issues with people discussing the points, or even speculating the motives but lets do without the attacks eh? Those that do are doing a diservice to the rest of the B3D posters that can generally behave themselves. Again, just because we operate a fairly open house don't take advantage of it - lets set the example.

 

[K.I.L.E.R]

Regardless, Dave wants a professional image for this site.

 

[martrox]

[(Martox's last reply springs to mind as one example).

Dave, while I understand what you are saying, I have to say that was said because Kyle saw fit to bring up something that was obviously said in private, for no reason except to be hurtful, period. Remember, it was Kyle himself that got personal to begin with (his attack on ET). The major reason that many have become abusive toward him is do directly to his abuse..... in other words, he has reaped what he has sowed. While I have disagreed with many(including Rev - just ask him!), I have never felt compelled to say anythig like this...... It just made me see red to see Kyle out Rev the way he did. If you feel I was wrong with what I said, then I beg you forgiveness, however, I still stand by my reasons for saying it.