[H] Benchmarking Future Ed.

John Reynolds said:
That's a nice way of putting it, but even with a gamer-oriented perspective Kyle is using an illogical piece of thinking (because a synthetic's DX9 shader performance fails to equate with an OGL game scores using vendor specific code paths it must be trash) to promote a blatantly obvious anti-Futuremark agenda.

I agree with your general thrust, and agree that it is tantamount to being anti-FutureMark (if we can use such a phrase without laughing at how ridiculous it is...;))

However, I think the anti-FutureMark stance is merely a sideline to the real agenda which is to help nVidia promote itself by way of discrediting any software which nVidia finds inconvenient from a commercial perspective. Note that the latest nVidia attack is on "fraps" (for goodness' sake) and that certain [H] staffers have taken to adding perplexing and baffling signatures such as "The Benchmark Slayer" to their posts....;) So, FutureMark is just one of many demons nVidia feels it must attempt to slay because it views such software as diametrically opposed to the ends of selling its hardware.

I think the chances of nVidia's success in this endeavor is roughly equivalent to the probability of us seeing a manned mission to Mars set foot on the Red Planet by this Christmas...;) For that reason it is odd indeed to see [H] staking the entire credibility of the site, something it has taken years to build, on such a silly set of notions like "frame-rate benchmarks are dead" and so on. [H] is likely to share whatever lack of success comes to nVidia by way of this campaign. It surely will not succeed.

Instead, what has clearly transpired is that the nature of benchmarks is definitely changing, but the change has nothing to do with eliminating frame-rates; rather, it has to do with adding as much emphasis on IQ as heretofore has been placed mainly on frame-rate counts alone. Yes, benchmarks are changing, but they are evolving into "frame-rate & IQ" benchmarks instead of being mere frame-rate counters. And that's what I see that is troubling nVidia--they'd just as soon everything revert to frame-rates only or else no one run any benhmarks at all. I can't see any difference between the position nVidia has taken and [H]'s position, unfortunately. Both are equally illogical, and equally improbable.

I also find it distasteful how it's never mentioned that those Doom 3 scores were derived from workstations supplied/built by Nvidia. So he's not only using a very illogical argument but he's also building it on a potential house of cards.

By far the site most impressed with the "Doom 3 preview" is [H.] It comes up in Kyle's editorial comments again and again. Most everyone else understood it for what it was: a contrived, carefully controlled PR event the main purpose of which was to promote nVidia's nv35 hardware. It was neither open nor fair nor objective in any sense of the word, and Kyle's pretensions otherwise have not taken root in sites other than [H.] His position that a pre-release game *demo* utilizing a vendor-specific path (and therefore itself not representative of the bulk of 3D games) is "A-OK", but a benchmark using generic DX9 shader code is an abomination, is really too bizarre to understand.

But so is nVidia's anti-benchmark crusade, which is just dumb. If nVidia's hardware was as it is represented by nVidia then the company would have nothing to fear from people using such software. IMO, nVidia is simply afraid of people using that type of software to discredit its marketing efforts. Either Kyle doesn't see that he's being used as a dupe, or else he intends it deliberately. What on earth would make anyone imagine that the only thing people are interested in, and should be interested in is how well a piece of hardware runs a pre-release Doom 3 preview demo that few people on earth have had any access to. If he thinks that this is the only avenue of interest people have in looking at 3D chips he is of course very much mistaken.
 
John Reynolds said:
I also find it distasteful how it's never mentioned that those Doom 3 scores were derived from workstations supplied/built by Nvidia. So he's not only using a very illogical argument but he's also building it on a potential house of cards.
Actually, IIRC, [H] was the only site to use their own machine with nV's HD, b/c nV's machine either never reached them or was damaged in transit.

Still, the demo was provided and sponsored by nV at its request, and it seems ATi was not given equal warning about the test. So the results aren't fully valid, IMO, especially now that I know there's a difference between app-requested and driver-forced AF, which is the difference between D3's MQ and HQ modes.

I agree with Kyle's assessment of this forum as overly vicious. Sure, we may know more than other forums, but who gives a flying Dutchman when all you read is how someone was paid off or spewing garbage, etc. A little more civility would suit this forum well. I'd rather have Kyle in here to debate with, even if I disagree with him. Chasing him out with rather rude remarks (like he was so easily able to find) isn't really doing anyone a service.
 
micron said:
An [H] forum member wrote...
Beyond 3D
You'll find many more informed opinions over at that site than you will anywhere.

Kyle wrote
Actually I think you can find more informed opinions in a Yahoo chatroom.

I like how your "experts" resort to name calling, conspiracy theories, picking apart hypothetical example, and simply discussing how they refuse to read the whole thing, but continue to comment on it like they know what the article states.

Here are some great quotes from your experts.

"Surprise! Surprise! Did he get $50,00 for this one? "

"Garbage, as usual "

"The real danger to Kyle with all of the nonsense he's spouting here is that people will begin to see how little he apparently understands about the very topics on which he opines... "

"That dood either need to stop doing drugs or start doing drugs because something is wrong with his brain."

"I'm just disapointed in myself for even reading the first page of that editorialmercial and giving that tool Kyle even 1 hit from me. "

"Well, I will not visit the site so I can't read the whole article. But based on the blurb that was copied, here is my translation: "

"He must be hallucinating or something. "

"And, this, children, is an example of just why drugs are bad for you........ "


I somehow am not as impressed by their expert opinions as you are. Sounds like fanboys riding their little radio flyer bandwagon around looking for someone to actually give them attention. But that is just my opinion.
Though Kyle doesnt post here anymore, he deffinately remains here with us. Hi Kyle ;)

YAY! He quoted me. :LOL:

How exactly are we fanboys? nVIDIA cheated and I am simply disgusted by it. When Ati cheated with the Quack thing I was equally disgusted.

Seems to me, f-an.bo.y = someone who doesn't agree with Kyle.
I don't agree with his article because the majority of points he makes are the points that nV have made. They are just re-worded to make them sound like it comes from him.
If the person reading the article is literate, they will be able to see that.

It seems to me that he is a copy-cat PR machine made by NV with the $$$ offered to him.
 
IMO, I think Kyles views in this article are slightly myopic and one of capitualation rather questioning whats actually good for the industry.

He correctly points out that games are coming along with different code paths, and he even suggests that there may be hardware specific titles – which will be an absolutely horrific situation for the PC gaming industry as it will end up in 4 (and fifth) place in the console race, and eventually drop off entirely.

Instead of capitulation to the situation shouldn’t we be singling out what we feel is for the better for the industry as a whole?

I’ve had a number of phone conversations with NVIDIA since this 3DMark issue started and their point of view is that demo’s and benchmarks such as 3DMark and Shadermark are not representative of games because they weren’t coded specifically with NVIDIA shaders in mind, but games on the other hand will be – by either the developer doing it off their own back or games that go through the TWMTBP campaign. This is the issue that is key to the approach to ‘Application Specific optimisation’ in benchmarks they appear to feel they have the right to change the shaders from those that were coded because this will actually reflect game code more. As I see it there’s two issues with this point of view.

First, what about those games that are not on the TWMTBP campaign? Do we really feel that every game will be on it? There are also a number of software houses that have been using 9700 as their DX9 development platform – what will happen when they attempt to put their code on GeForce FX hardware? I’m hearing talk of one AAA title that is having these very issues at the moment.

The second issue is whether creating an architecture that requires lots of app specific tweaking is a good thing or not. For instance, had Doom3 not required five different code paths you might ask if the game were not here now? If developers are spending time tweaking shaders for NVIDIA’s hardware could this time not be better spent elsewhere – is it stifling creativity not having two generalised shading architectures? You may ask if that is an issue for those games going through TWMTBP as NVIDIA would be doing the work themselves, but if that’s the case would the resources at NVIDIA, that were paid for by you, not be better spent up front in creating a generalised shader hardware architecture such that it doesn’t need a large set of resource at the backend making software work well with the hardware?

These are the two points that actually need addressing IMO. Synthetic benchmarks are just there showing where there are obvious deficiencies in hardware and at the very least they are highlighting the areas that developers need to code around in order to gain acceptable performance on the NV3x architecture in comparison to the R3x0 architecture.

Of course, Kyle has now taken a stance, and his stance seems to be one that NVIDIA’s PR has been talking about since the introduction of the FX series. The danger for Kyle is that PR is only out for the good of the company at that time – we all know that PR sentiments flip-flop according to what their hardware does at any one point in time in relation to their competition. Should NV40 (or NV50) turn up and suddenly have a much more generalised, high performance shader architecture that doesn’t need lots of tweaking and mixing of precisions to get good performance what’s the betting that the noise about synthetic benchmarks will die down again?
 
These are the two points that actually need addressing IMO. Synthetic benchmarks are just there showing where there are obvious deficiencies in hardware and at the very least they are highlighting the areas that developers need to code around in order to gain acceptable performance on the NV3x architecture in comparison to the R3x0 architecture.

Dave, synthetic benchmarks can not only be used to detect deficiencies but to compare raw performance of different architectures. Optimising, and changing the way the synthetic benchmark operates/runs/etc... is manipulative and sneaky.

Kyle and nVIDIA say that more and more games will come out having shaders specifically optimised for the NVxx line of cards but we don't even have a massive number of games coming out using shaders. Give me a large list of all the games coming out using shaders to render above DirectX 7 style graphics?

Don't say Doom 3 either. ;) :)

That is also another reason why nV's/Kyle's point is rubbish.
 
The issue I have wrt Kyle specifically has to do with the fact that his opinion of Beyond3D is based on non-Beyond3D staff (i.e. open foum participants') expression of opinion. He is saying, basically, that "I have no respect for Beyond3D because their forum particpants are flaming me".

If he could address what Beyond3D -- the site, its staff, its partners -- are doing wrong, I wouldn't have a problem with him. But he is treating B3D's forum participants' opinions as officially representative of Beyond3D. That is wrong.

He has been "in the business" a long time -- forum participants != represent Beyond staff.

Lastly, NVIDIA wants to be a monopoly. I don't blame them... I'd try to do the same... it's business.... it's about making money. What Kyle doesn't appear to understand is exactly as dave stated above -- he has failed (or appear to, based on his numerous expressions of opinions) to take into consideration the potential of a company out to dominate and monopolize.
 
Reverend said:
The issue I have wrt Kyle specifically has to do with the fact that his opinion of Beyond3D is based on non-Beyond3D staff (i.e. open foum participants') expression of opinion. He is saying, basically, that "I have no respect for Beyond3D because their forum particpants are flaming me".

If he could address what Beyond3D -- the site, its staff, its partners -- are doing wrong, I wouldn't have a problem with him. But he is treating B3D's forum participants' opinions as officially representative of Beyond3D. That is wrong.

He has been "in the business" a long time -- forum participants != represent Beyond staff.

Kyle must have been real angry when he made his statement. People make wrong choices when they are angry. I do as well.
Just ask John Reynolds about one of my PMs about Sage. I was threatening legal action that's how angry I was. Of course I retracted what I said after I calmed down.

Lastly, NVIDIA wants to be a monopoly. I don't blame them... I'd try to do the same... it's business.... it's about making money. What Kyle doesn't appear to understand is exactly as dave stated above -- he has failed (or appear to, based on his numerous expressions of opinions) to take into consideration the potential of a company out to dominate and monopolize.

You bastard. :LOL: ;) :D

--------------------------

On a more personal note, I have nothing against Kyle. He can only do damage to himself.
 
DaveBaumann said:
The second issue is whether creating an architecture that requires lots of app specific tweaking is a good thing or not.

The bizarre thing here is that it is not so long ago that NV were hyping the XBox because it was easy to develop on in comparison to PS2.

Now, it appears almost as if the new NV3X chips are the "PS2" of the PC graphics world as they are more convoluted to program for than other chips.

I would have thought that very few game developers are happy with the current state of affairs as regards the need to program for multiple-precision due to NV's design decisions. Just think how much easier it would be if NV had gone for high-performance at a single precision (whether 16fp, 24fp or 32fp) - it would just be a case of write the shader and let the hardware do the rest as opposed to all the hullabaloo about Cg etc.
 
we all know that PR sentiments flip-flop according to what their hardware does at any one point in time in relation to their competition

For example hyping 32-bits and image quality at one time, then one generation later pushing the power of T&L and framerates at 640x480x16 ? :p

Very nice summary, Dave.

TWIMTBP program is one of the worst things that could happen to PC gaming... Crappy XBox ports only running on NV hardware for lack of testing being another one.
 
DaveBaumann said:
First, what about those games that are not on the TWMTBP campaign? Do we really feel that every game will be on it? There are also a number of software houses that have been using 9700 as their DX9 development platform – what will happen when they attempt to put their code on GeForce FX hardware? I’m hearing talk of one AAA title that is having these very issues at the moment.

The second issue is whether creating an architecture that requires lots of app specific tweaking is a good thing or not. For instance, had Doom3 not required five different code paths you might ask if the game were not here now? If developers are spending time tweaking shaders for NVIDIA’s hardware could this time not be better spent elsewhere – is it stifling creativity not having two generalised shading architectures? You may ask if that is an issue for those games going through TWMTBP as NVIDIA would be doing the work themselves, but if that’s the case would the resources at NVIDIA, that were paid for by you, not be better spent up front in creating a generalised shader hardware architecture such that it doesn’t need a large set of resource at the backend making software work well with the hardware?

On top of that, the thing that bothers me about optimising for a specific architecture is what happens when the next generation of cards comes around and the architecture changes again?

Say you own an NV30/35 now, and everything is great because developers of most games are encouraged to write shaders with the architecture of your card in mind.

Sin months later, NV40 is released and developers are suddenly required to change the way they write shaders and start optimising for NV40. NV3x cards are left forgotten, and you find that your state-of-the-art card which should last you a couple of years has become obsolete in six months, not because games have suddenly become more complex but because your card isn't being optimised for any more.

Of course, this scenario is another win for the IHVs, because it means buying a new card ever more frequently, but it leaves room for a lot of unhappy customers.
 
...ten, and you find that your state-of-the-art card which should last you a couple of years has become obsolete in six months, not because games have suddenly become more complex but because your card isn't being optimised for any more.

That's why we have the ARB path. :)

Game devs should write for a standard codepath and IHV should optimise for that code path.

If the NV3x line was twice as fast on the ARB/2 path than the R3xx line, do you really think nVIDIA would be asking for optimisations? Gamedevs wouldn't bother, nVIDIA would optimise their drivers and start touting "we have the industry standard bleh bleh bleh...".

Currently they are touting, we have future games optimised for our cards.

If you take a close look at it, the NV3x line of cards NEED an optimised code path otherwise they will be falling behind in performance and more so in quality. IE: Doom 3 ARB2 path on NV3x vs R3xx.

Which will drain consumer confidence and shake the company well and truly into a large "oh oh". :LOL:

The sad thing is, the R3xx line of cards compete (when NV3x uses optimised path vs R3xx using ARB path) and even triumph over the NV3x line in performance (when both line of cards are on equal ground) and decimate in terms of IQ (regardless). :)
 
Reverend said:
If he could address what Beyond3D -- the site, its staff, its partners -- are doing wrong, I wouldn't have a problem with him. But he is treating B3D's forum participants' opinions as officially representative of Beyond3D. That is wrong.

Heh, I think he did that here Rev:

Kyle said:
Anthony Tan came to me a couple of months ago and asked for the lead VidCard reviewer’s job here at HardOCP. He was turned down. I think that has lead to some of his recent animosity towards us.

First it was Extremetech that was pissed cause they didn't get to bench doom3 like he did. Now you disagree with him because you didn't get the Lead VidCard reviewers job. The guys got an ego a mile wide. And you wonder why he gets dumped on here Russ?
 
Kyle is a total ass.... The more I hear from him, the more I realize just what an ass he is..... It's one thing to disagree with someones point of view, it's another to use ones position to abuse others.....
 
I think where Nvidia (and Kyle in their wake) have gone wrong is that they have forgotten their history. When 3D cards were first arriving, every chip manufacturer had their own API, and a programmer had no choice but to hit the metal. Even then, there was no guarentee that your content would look the same from card to card. Developers hated that, and Direct X eventually became the PC gaming API of choice because it meant there was one basic API where content looked pretty much the same everywhere. This was only possible via the massive power of the Microsoft desire to take over everything and provide a basis for their X-Box development.

OpenGL only survived as a gaming API on the PC through the Mighty Power of Carmack, and his insisitence on cross platform development, but with standardisation, we are left with these two main APIs where if everyone sticks to the rules, we should see developers with one of these two main codepaths, and content that looks pretty consistent on any piece of hardware.

Now along comes Nvidia (trailing PR extensions like HardOCP) with it's inferior NV3x range, that can only compete with customised paths, low precision shaders, etc, and tells us that Nvidia is such a driving force, that all developers are going to rush back to the old model of coding a customised path for every bit of hardware out there. Unlikely.

There is a reason that the old model went by the wayside, and that is because developers and publishers hated it. Developers hated the hassle, publishers hated the increased cost and development time. That's why it was replaced by DirectX and OpenGL. Even Nvidia realise this, as they are still pushing CG as a way of hiding their attempt to introduce an "NVGLide" API.

Nvidia think (despite being the best part of a year behind by the time they get any true DX9 hardware to customers) that they are big enough to pull the developers back towards custom codepaths in order to make NV3x palatable. Either this is more PR spin that even Nvidia doesn't believe will happen, or that (even worse) Nvidia really believe they can convince developers to go backwards.

This indicates to me that Nvidia still cannot admit their problems to themselves, and until they do so, they have no chance of fixing these issues. They simply do not perceive these things as in need of fixing. They are compeltely out of touch, as shown by thinking things like FXFlow would be a good idea, and would be accepted by the market. Nvidia are in complete denial, their PR running out-of-control (as that is all they have at the moment) and people like Kyle are simply seen (by both Nvidia PR and the more savvy technical users) as an extension of that marketing campaign, and continued state of denial from Nvidia.

I feel sorry for Kyle. He's been taken for a ride by Nvidia. IMNSHO, he's been completely used. One day he might wake up and realise that, but to be honest, I think it will take a very big man to admit to himself the corner he has painted himself into. Unless he really is just a PR bunny and thinks that there is nothing more important than marketing and website hits, he's going to feel very disappointed one day.
 
jjayb said:
Heh, I think he did that here Rev:

Kyle said:
Anthony Tan came to me a couple of months ago and asked for the lead VidCard reviewer’s job here at HardOCP. He was turned down. I think that has lead to some of his recent animosity towards us.

First it was Extremetech that was pissed cause they didn't get to bench doom3 like he did. Now you disagree with him because you didn't get the Lead VidCard reviewers job. The guys got an ego a mile wide. And you wonder why he gets dumped on here Russ?

LOL Kyle offered me that position in the spring of '01. I turned him down. The funny thing, at least to me, is that Anthony would've done a helluva better job than I would've.
 
Some time ago, Britney Spears approached Kyle about being his lead love monkey boy toy sex kitten. But Kyle knew that over the next 24 months that Nvidia was moving the industry away from the open Britney love monkey boy toy sex kitten model, and towards a more proprietary one. Kyle felt that while having Britney as his lead love monkey boy toy sex kitten might be nice in the short-term, it would not serve you, the consumer, well in the long-term. That's why Britney does Pepsi commercials now, when she knows that Kyle is solely a Jolt man.
 
WaltC said:
However, I think the anti-FutureMark stance is merely a sideline to the real agenda which is to help nVidia promote itself by way of discrediting any software which nVidia finds inconvenient from a commercial perspective. Note that the latest nVidia attack is on "fraps" (for goodness' sake) and that certain [H] staffers have taken to adding perplexing and baffling signatures such as "The Benchmark Slayer" to their posts....;) So, FutureMark is just one of many demons nVidia feels it must attempt to slay because it views such software as diametrically opposed to the ends of selling its hardware.

I've had that in my Sig since before all this stuff started

I'm a fan of Buffy the Vampire Slayer, and thats the best thing I could think of that fit that tag line, even though it really doesn't make much sense, heh

it doesn't relate to any of this

http://www.hardforum.com/showthread.php?s=&postid=1024948932#post1024948932
 
Brent said:
WaltC said:
However, I think the anti-FutureMark stance is merely a sideline to the real agenda which is to help nVidia promote itself by way of discrediting any software which nVidia finds inconvenient from a commercial perspective. Note that the latest nVidia attack is on "fraps" (for goodness' sake) and that certain [H] staffers have taken to adding perplexing and baffling signatures such as "The Benchmark Slayer" to their posts....;) So, FutureMark is just one of many demons nVidia feels it must attempt to slay because it views such software as diametrically opposed to the ends of selling its hardware.

I've had that in my Sig since before all this stuff started

I'm a fan of Buffy the Vampire Slayer, and thats the best thing I could think of that fit that tag line, even though it really doesn't make much sense, heh

it doesn't relate to any of this

http://www.hardforum.com/showthread.php?s=&postid=1024948932#post1024948932
I'll vouch for Brent on that one, he has had that sig for a long time before this whole thing came out. It has nothing to do with the recent benchmarking fiascos, although I must admit that I almost fell over laughing when I realized the irony of it yesterday. :LOL:

Brent, how? I mean, you KNOW better. :( I might just be a simple enthusiast/fanboy, but even the bigbrains around here are against what [T]ardOCP is doing!

It's going to stop and it's going to end, houses of cards always seem to fall; what happens then? And if it doesn't fall/fail, are you going to be proud for the part you had in bringing in the bad old "4 different codepaths for a game" days again?

I like you Brent, I really do and I've admired/respected the work you have done at [T]ardOCP and was hoping that you'd influence them to improve their standards...instead it seems your own are slipping.

Best of luck with whatever happens Brent, but I'm currently quite disappointed. :(
 
Back
Top