Kyle's new thread @[H]

[micron]

At least I got to pop off the first question....
http://www.hardforum.com/showthread.php?s=&postid=1024834575#post1024834575

 

[demalion]

Wow, I love the technical critique of 3dmark 03 he provides.

Or not.

How can someone call themself objective for so directly parroting a stance hand fed to them by a party with a vested interest in that stance, and providing support for that stance by"logic" that completely fails to provide any remotely technical explanation for the conclusions presented?

...
Futuremark asks the question, "Can 3DMark03 be used as a reliable benchmark for DirectX 9 generation graphics cards?" And Futuremark answers, "Yes, with the new 3DMark03 build 330, it can." Logically, any version of 3DMark03 prior to this build mentioned above has not been a reliable benchmark by their own admission. I personally have never thought it to be a reliable benchmark, albeit for other reasons than exposed here. I think it does more to harm the community than benefit it.
...

This is seems to be the only logical support presented on his stance in the thread so far.

Here is a sample of the conclusions based on this logic:

...
So to directly answer your question, I am not sure how one can cheat at altering the score of a benchmark that obviously has no value to begin with.
...

I think he's said he had a laugh at some of the comments in this forum. I really think he should pay more attention and go back and read some of those comments again. Hey, but I'm not a webmaster, what do I know? If you run a website, maybe you stop needing things like reasoning to support your opinions.

 

[Rodéric]

That's like : «tout ce qui est rare est cher; or un cheval bon marché est rare; donc un cheval bon marché est cher» (called a Sophisme in french)

rough translation :
"Everything rare is costly, a cheap horse is rare, so a cheap horse is costly..."

 

[Ailuros]

*yawn* to the target of this thread (not the thread here at B3D itself).

 

[Doomtrooper]

Simple way to fix it, don't go there...I have removed them, and he can spread rubbish to people that want to listen to his utter crap.

 

[WaltC]

Kyle obviously doesn't see that his argument is circular and illogical. The only reason 330 is "better" than 320 as a DX9 benchmark is because nVidia's current driver set cannot cheat 330...

Kyle is pretending that the benchmark was worthless *before* nVidia cheated it and his prejudice has blinded him so that he cannnot see that nVidia cheating is what made it 320 "worthless" in the first place. He apparently doesn't care if nVidia misrepresents its performance (what's so hard about understanding a work-done/time = performance formula?)

Kyle either completely misunderstands the issue, or else he's hopelessly a nVidia shill. Haven't made up my mind which yet...he's either being played like a violin by someone in nVidia or he just doesn't get it, or he gets it and is as guilty as nVidia in perpetrating this fraud.

 

[YeuEmMaiMai]

Kyle is just

1. someone with NO integrity
2. someone that really does not understand what is going on
3. someone that has a good friend working at nVidia
4. someone that contrary to what he says DOES NOT CARE about the consumer getting short changed unless it is someone else besides nVidia.
5. someone who considered the quack thing a cheat even though he KNOWS that the bug was present in at least 7 to 13 previous driver revisions.
6. someone that has the intelligence of a two year old.
7. someone who only cares about page hits for his website

 

[epicstruggle]

1. If an IHV cheats to gain performance on a benchmark (3dM03/SS/Q3/UT2k3/...) what do you feel is the proper responce from [H]? What if the allegations of cheating cant be confirmed by [H] due to the lack of proper tools/knowledge, but have been by other sites(ET/B3d)?

2. Do you feel it was necessary to allege that ET had other motives because they did not get to benchmark D3, when it now has been confirmed that NV cheated to inflate their score?

Two days after Extremetech was not given the opportunity to benchmark DOOM3, they come out swinging heavy charges of NVIDIA intentinoally inflating benchmark scores in 3DMark03

3. FM responded that their type of benchmarking suite is necessary for this type of cheat catching, ie how easy would it be for you to catch an IHV if their cheating at a timedemo that also runs on a "rail"?

1. Answered above...

2. To nitpick, I don't think we "alleged" anything. Our statements were statements of fact and stated there to hopefully make you see the situation for being multifaceted, which I think it is.

3. This statement is ludicrous at best as Futuremark is simply trying to make excuses of why they are needed. A good reviewer with the tools available in Quake 3 can identify this behavior in question.

As to Question 3, is it possible to cheat at a timedemo so that even a "good reviewer" could be fooled?

later,

 

[Dave Baumann]

When a timedemo is running 200-300 frame a second, I'll be damned if I could see things such as frame being dropped or other issues.

 

[Mark]

When a timedemo is running 200-300 frame a second, I'll be damned if I could see things such as frame being dropped or other issues.

That's the exact thing I thought as well.

I'm no programmer, but after seeing what nVidia did to 3Dmark03 I wouldn't think it would be too hard for them to enable some cheats (discard some textures, turn off trilinear/anisotropic, etc) when running a common demo or game in benchmark mode.

 

[Rookie]

In the Nvidia statement letter I received,Nvidia even suggest to read Kyle 's benchmarking right article:

"If you need some additional 3rd Party support - the following editorial from HardOCP is a perfect response:

http://www.hardocp.com/article.html?art=NDMw"

 

[Dave Baumann]

What statement letter is that?

 

[Doomtrooper]

A scan/screen capture would be in order

 

[Reverend]

So to directly answer your question, I am not sure how one can cheat at altering the score of a benchmark that obviously has no value to begin with.

Brent must feel pretty upset. He has used ShaderMark a few times in his reviews. I wonder if ShaderMark has "value to begin with" compared to 3DMark03.

I must admit I didn't read the [H] thread. Did Kyle respond to the question posted in the very first response in that thread? What did he say, if he did respond?

I think everyone should just stop posting Kyle's opinions on this matter at our forums -- just email him saying that he should think seriously about telling his staff that writes video card reviews for his site to stop visiting B3D and getting more interested in 3D Technology and just hire those that are solely interested in running currently available games, which more-or-less mirrors his opinions. He may just have a more successful and less-debatable site if he does this.

I wonder what Brent would decide if Kyle gives this "ultimatum".

Note that I believe "games only, dude!" websites have as much a place in this review industry as "technology only, dude" websites... although I'd like to think B3D has a good mix of both. Kyle's stance on the current NVIDIA-3DMark03 issue seem to be "games only".

My opinion of course, not B3D's

 

[John Reynolds]

I must admit I didn't read the [H] thread. Did Kyle respond to the question posted in the very first response in that thread? What did he say, if he did respond?

I just forced myself to read the thread. It's like poking an existing bruise to see if it's still sore. . .you just can't resist. 8)

Actually NVIDIA has said nothing directly to me on this issue as I have not discussed it with them. Unless you count circumstantial evidence, I don't think you have proof that it is not a bug. It will be interesting to see what comes out of this. Also I might add even after ATI got caught red-handed with the Quack driver issue their final official response was that it was a bug. ATI's Rick Bergman, Senior Vice President, Marketing and General Manager, Desktop, told me that directly to my face in a meeting. It was my opinion that they were cheating and still is.

I'm sorry but this is reaching such levels of <bleep> that it's becoming rather amusing to me.

 

[Reverend]

BTW, I'd thought it important/relevant to say the following.

For the longest time, while I ran my own Pulpit webiste, I refused to provide 3DMarkXX analysis in my video card reviews because I had wanted Pulpit to be "gamers-only" website. That does not mean I am not interested in 3D technology, which is what 3DMarkXX is about (I think some forget that 3DMarkXX is not about games but 3D technology) -- it's just that I have a very definite focus for my personal website. I could just as well start another Pulpit that focussed more on tech than gaming, but I was just too lazy and never had the time

I don't think I'd be far wrong to say that many of the Pulpit's "enthusiasts" (i.e. those that agreed with the website's focus -- not my personal focus... there's a difference) migrated to B3D when I joined this site. And as a result, their interest expanded, instead of changed.

Kyle, IMO, doesn't seem to know what HardOCP's -- his site -- focus is. Or he is having a hard time dealing with his vid card review staff's expansion in interests. Or he just doesn't know what his own personal focus is. Other than to try to appeal to widest possible audience in his website venture while trying to have some sort of respectable reputation by incorporating the "tech route" as evidenced by Brent's and Sean's recent reviews that incorporate synthetic benchmarks (which, really, when it comes to existing games, has no "value to begin with").

Again, my personal pinion

 

[Kristof]

So to directly answer your question, I am not sure how one can cheat at altering the score of a benchmark that obviously has no value to begin with. The real losers here are the persons that actually think that the 3Dmark03 score really means something. The overall 3Dmark03 score is absolutely useless for gauging anything about video hardware when it comes to evaluating real world gaming situations. BUT, if it was a real game and NVIDIA was pulling this BS, we would be happy to see their asses publicly whipped. If NVIDIA goes there, I think we will obviously know who is smoking the hallucinogens then.

Note how he talks about the overall score of 3DMark, I think everybody agrees that the total score as reported by 3DMark has no real value - the detail scores however are very valuable and there is no way to possibly claim that they are not useful. Afterall when do you expect that a game will come out with the shader load available in 3DMark today and decent benchmarking functionality ? They keep talking about Quake3, which for todays graphics hardware is pretty much turning into a CPU test (how low can your driver overhead go ?).

In a real game how is he going to check each shader ? Can he run every game using the ref rast so he can judge if the correct accuracy is being maintained and not some hacked looking similar shader ?

All in all 3DMark is a very valuable test set, the score is just a number which indicates that higher is better and it satisfies the most basic user that only cares to check if his system is performing roughly as it should - it also satisfies the tweak freaks so they can battel for the highest scores.

K-

 

[Dave Baumann]

I'd suggest its perfectlt reasonable for their reviews to be "Gamer only" reviews, however if they want to go down that route they shouldn't at all tout the next generation features that are being advertised with each new generation of board. Synthetic test such as 3DMark, and others, are the only ones that can provide us with any clue as to the true relative performance using next generation features - if nobody had used 3DMark or Shadermark etc. would anyone have realised that the actual DX9 abilities of ATI's hardware is better than that of CineFX's?

 

[Reverend]

Another "BTW".

I wish that you guys would stop posting a thread like this in this particular forum. In fact, I wish you guys would simply stop posting threads like this here per se. At the risk of sounding "elitist", I'd really appreciate it if we could return to the old, old Beyond3D Forum, where almost all threads are about 3D technology and that if you want to sound off about other website opinions, do it at other forums. Again, sorry if this sounds kinda "elitist" or pompous.

I'm not going to decide what to do with this thread... John can have the pleasure

 

[John Reynolds]

Another "BTW".

I wish that you guys would stop posting a thread like this in this particular forum. In fact, I wish you guys would simply stop posting threads like this here per se. At the risk of sounding "elitist", I'd really appreciate it if we could return to the old, old Beyond3D Forum, where almost all threads are about 3D technology and that if you want to sound off about other website opinions, do it at other forums. Again, sorry if this sounds kinda "elitist" or pompous.

I'm not going to decide what to do with this thread... John can have the pleasure

Agreed. I'll let this one run its course but in the future any such threads will probably be moved to the General Discussion forum.