Firingsquad Fires Back

In other words, ATI’s card isn’t going to suddenly come out ahead of NVIDIA because the timedemo didn’t include physics or AI.
That's where FS loses all credibility in my view.

Until this assertion is proven for each individual game - something that I've never seen - I will always trust real gameplay run-throughs over "timedemos".

Timedemos that don't use AI or physics (etc.) can be good for testing graphics cards - but only when those tests are not comparing ATI and NVidia drivers. Hence the B3D style of review.

Jawed
 
I do recall quake 4 showing ati a bit ahead with the time demos but real world results showed they are a bit slower than nvidia.
 
Kyle said:
We have moved away from “benchmarksâ€￾ fully, real-world or not. Firing squad fails to even understand that we are not "benchmarking."
a dictionary said:
benchmark: A standard by which something can be measured or judged.
Er... well, at least this gives me something to do tonight.
 
They survive by the canned benchmark. A well trained monkey can pull of that sort of “hardware journalism.â€￾ So why do they continue to test like this and now find it convenient to attack real world testing? That is simple to answer and it all comes down to money. Real world testing is very expensive and time consuming. Canned benchmarks cost nothing in comparison. Real world testing is starting to develop a wider fan base as more of you are seeing the true value in it. Canned benchmarking is showing its age and its value is waning as gaming and applications are changing and becoming more complicated. The more people that slip away from wanting to view their pages of canned benchmark graphs means less money for Firing Squad.

It seems to me the "economic interest" argument cuts both ways. To the degree [H] can jawbone the community into more resource-intensive methodologies, the more resource-blessed sites (like [H]) will be at a competitive advantage.

So I think the debate needs to be on the merits, rather than "That other fellow wants to make money!". Well, duh. :LOL:
 
Interesting, I think we are seeing the two extremes here.

Personally, I feel that the "canned" benchmarks do have some merit, as their single focus can often enlighten users and reviewers as to why some hardware behaves as it does. At the same time we do not sit for hours in front of our computer watching 3D Mark for pleasure. At the same time though, FS has some good points in that Kyle's idea of playable is not the same as ours, and if some user in a deathmatch/competition needs the extra CPU cycles while cutting down the bottleneck of the graphics card by playing at 800x600 with the eye candy turned off, but getting 60 more fps in a modern game by going Conroe over an X2 can be just the edge they are looking for.

The name calling should stop though. We all have different philosophies on testing, and it is good to respect those. You may not agree with it, but you don't need to pan those of us that do use synthetics, or run specific demos, or *gasp* run games at lower resolutions to see how CPU's scale so we can get a better idea of performance when the next gen of video cards come out and prove to be less of a bottleneck.

Heh, just my opinion though.
 
I don't know, both have merits. Why not use both? With GPU's -- why not offer Apples-to-apples comparisons and then offer subjective real-world settings, too. Why does it have to be one or the other?
 
Look, up in the sky! It's a benchmark! No, it's a real world test!

This sounds like a job for ... a Prof. of Sophistry. Baron, you're up. Oh, you already called it. Right, then.

Seriously, this whole thing has the smell of Rydermark: internet sniping with minimal editing to attract attention. Journalism at its finest! Always interesting to see the philosophers (Kyle) following the student (Fuad) into the din of short reflection and grand pronouncements. At least Jakub stuck a "Rant" in front of his piece.

Or maybe I got the master/student titles wrong. Eh, whatever.
 
I know we can all try to take "there is room for both" approach, but on this particular matter, FS wins and Kyle loses. The facts of the matter are:

1) Kyle used "real world gaming" exclusively for his Conroe review, while only a month ago for AM2 review he used low-resolution CPU sensitive benchmarks. This, IMO, was done solely to prove his non-point about Conroe's lack of performance for games and is, at the very least, a somewhat of double standard and creates and appearance of (not so?) hidden agenda.
2) His belligerent tone towards both the rest of the community and Intel was completely uncalled for.
3) As pointed out million times, his methodology has serious flaws that may do a disservice to his readers. I will not reuse my car analogy from the other thread, but not only does it not provide the reader with any information other then how his/her graphics card behaviors under the given set of conditions, it also assumes that the reader has the same hardware and preferences as Kyle.
4) FS does suggest that both “real worldâ€￾ and CPU-limiting benchmarks should be used while Kyle takes the “my way or you are an incompetent Intel crony who can’t afford to things properlyâ€￾ approach.

BTW, I am looking forward to the "all you need is an Athlon XP 2400+" conclusion that from now on will inevitably be included in the every review of new CPUs at [H] … right?
 
[H] has always been a dumb site..this just takes a bit further.
They're one of those proud to be ignorant types.
Kyle= big dumb pompous prick.. always has been and likely always will be.
 
Jawed said:
That's where FS loses all credibility in my view.

Until this assertion is proven for each individual game - something that I've never seen - I will always trust real gameplay run-throughs over "timedemos".

Timedemos that don't use AI or physics (etc.) can be good for testing graphics cards - but only when those tests are not comparing ATI and NVidia drivers. Hence the B3D style of review.

Jawed
I thought their argument was that because there is no AI or physics there is just less strain on the CPU which wouldn't shift a clear advantage that one card would have over another in the same timedemo with AI and physics turned on. They're comparing timedemoes with AI and physics (like Quake4 and UT2004 that they mention) vs timedemos without that. It's not as if they're arguing timedemos are just as good and trustworthy as real gameplay tests.
 
each of nV and ATi drivers have different cpu overhead so time demos that doen't account for CPU usage will make a noticable difference.
 
Honestly, Kyle's editorial left a sore taste in my mouth, mainly because I sensed a "the world revolves around [H]" mentality - can't put it in a better way. It's one thing to defend your views and the way you work and measure performance, and a totally different ballgame to start attacking others and accusing them of creating reviews spanning lots of pages merely for the $$$ (which could be true, but it's totally irrelevant to the point that the editorial wants to address).
 
Razor1 said:
each of nV and ATi drivers have different cpu overhead so time demos that doen't account for CPU usage will make a noticable difference.
This has become surprisingly easy to test.
 
Razor1 said:
each of nV and ATi drivers have different cpu overhead so time demos that doen't account for CPU usage will make a noticable difference.
I hadn't thought of that. Would it change the order of 2 cards?
 
If [H] is going to do benchmarks at only high resolutions, I think it'd be better for them to show a graph of the framerates and declare the winner by which one spent a great amount of the time above 60fps. Or which one had the more stable framerate.
 
XSBagage said:
I hadn't thought of that. Would it change the order of 2 cards?

Depends on the engine, but if the margin is within 5 FPS it possible could. But as Baron has said its benchmarkable now. FS's and [H] methods strive to show different aspects of hardware both benchmark methods are relevent and have thier own merits. I don't support the [H] article becuase one paticular benchmark or set of tests isn't better then another since each type of test suite shows different aspects.
 
Last edited by a moderator:
Back
Top