Shootouts in every B3D p/review?
- Thread starter Reverend
- Start date
- Status
And who runs the site.
NO! NO! NO! NO! It's stupid to try to compare a product that's available to one that isn't. You have no way of knowing IF the GFF....um.....product you are previewing is really the one that will be for sale this Christmas at CompUSA..... well, maybe by this Christmas....... 
I know that ATI was roundly booed when they first previewed the R300, because they didn't want scores published... But they were right, as those scores "may" have had no bearing on the retail product.
Want to compare products? Wait till they are available to all, then compare apples to apples (No, not a "Mac" pun)
I know that ATI was roundly booed when they first previewed the R300, because they didn't want scores published... But they were right, as those scores "may" have had no bearing on the retail product.
Want to compare products? Wait till they are available to all, then compare apples to apples (No, not a "Mac" pun)
I'm going to have to abstain from selecting an answer till I have more information. It all depends mainly on what you believe your target audience is. Do you believe your target audience are techies or gamers looking for purchasing decisions? Personally I love reading reviews that completely disect technology, but in the back of my head I'm wondering how it will compare to the competition because eventually I'm going to want to use that knowledge to make an informed decision. I suspect a lot of people are like that too, but how many are like that?
Another thing to consider is when are you recieving these products and what kind of products are they? The timing is definitely going to determine if your first article on the product is actually a technology preview, technology review, technology comparison, product preview, product review or product comparison.
Once you've determined your audience, the easier it will be to decide what to do above.
Food for thought.
Tommy McClain
Edit: Added a word
Another thing to consider is when are you recieving these products and what kind of products are they? The timing is definitely going to determine if your first article on the product is actually a technology preview, technology review, technology comparison, product preview, product review or product comparison.
Once you've determined your audience, the easier it will be to decide what to do above.
Food for thought.
Tommy McClain
Edit: Added a word
Joe DeFuria
Legend
I also agree that the question is basically worded wrong.
I'm a big proponent of shootouts, but I DON'T feel we need to see them in every p/review.
Having said that, I feel that looking at technology in a vacuum, per say, only tells maybe 1/2 the story. Half of the REAL INTERESTING bits come from comparing competing technologies. I think we all want to know the "pros and cons" of whatever architecture is being reviewed. Problem is, by definition and "Pro" or a "Con" means it is relative to SOMETHING. And I think the most relative comparison to be made, is one where you compare the technologies competing for the same market.
In short, I think FIRST having an article that is a "tech review" of one product in isolation is perfectly valid...but it only tells half the story.
Now, as a consumer, I would RATHER see shoot-outs of actual products. (Take two acutal competing products off the shelf, and run "apples to apples" benchmarks, etc.) However, I understand that this site isn't particularly geared particularly to consumers needs, so that might not go with Dave's vision.
However, even if a "product" shoot-out doesn't fit this site's goals, a technology shoot-out certainly does, IMO. You would still going to be running many of the same tests as a "Product" shoot out (though many more synthetic tests), but the focus will not really be on which one gets the best FPS or best image quality....but WHY the FPS results are what they are, and WHY the image quality differs.
There's a big difference there. But regardless of whether it's a "product" shoot-out, or a "technology" shoot-out, I think they are more or less mandatory to get a complete picture of the technology in general.
I'm a big proponent of shootouts, but I DON'T feel we need to see them in every p/review.
Having said that, I feel that looking at technology in a vacuum, per say, only tells maybe 1/2 the story. Half of the REAL INTERESTING bits come from comparing competing technologies. I think we all want to know the "pros and cons" of whatever architecture is being reviewed. Problem is, by definition and "Pro" or a "Con" means it is relative to SOMETHING. And I think the most relative comparison to be made, is one where you compare the technologies competing for the same market.
In short, I think FIRST having an article that is a "tech review" of one product in isolation is perfectly valid...but it only tells half the story.
Now, as a consumer, I would RATHER see shoot-outs of actual products. (Take two acutal competing products off the shelf, and run "apples to apples" benchmarks, etc.) However, I understand that this site isn't particularly geared particularly to consumers needs, so that might not go with Dave's vision.
However, even if a "product" shoot-out doesn't fit this site's goals, a technology shoot-out certainly does, IMO. You would still going to be running many of the same tests as a "Product" shoot out (though many more synthetic tests), but the focus will not really be on which one gets the best FPS or best image quality....but WHY the FPS results are what they are, and WHY the image quality differs.
There's a big difference there. But regardless of whether it's a "product" shoot-out, or a "technology" shoot-out, I think they are more or less mandatory to get a complete picture of the technology in general.
As to "shootouts" I would hope the answer is no. Beyond3d has always tried to show the benefits/downsides strengths and weaknesses of a card without getting into a benchmarkfest. Now, once a preview/review is done, taking a closer look at competing products may be a good followup article ...
...I think shoot outs are fine. Just when they are done and posted makes little to no differance. I would love to see nvidias NV30 vs the R350 though rather then the R300 vs the NV30. Since the NV30 isn't really selling on mass yet and you can expect that when the R350 is launched it should hit the market in relatively the same time frame they are more direct competitors. Maybe in the same shootout you could include the R300 or whatever even the NV34-NV31 and so on so that everyone can see the spread of the newest technology/cards compared. Rather this then comparing the NV30 to the 6+month old R300. The shootout would be best if it included a wider variety of the newest tech on the market not just the NV30 vs 6+month old card.
Also I think some shootouts are fine. But should Beyond3D adopt a policy of doing vast numbers of them then IMO it would taint the sites credibility in some strange way. The technical reviews/previews are great. Lots of shootouts = not so great. JMO.
Also I think some shootouts are fine. But should Beyond3D adopt a policy of doing vast numbers of them then IMO it would taint the sites credibility in some strange way. The technical reviews/previews are great. Lots of shootouts = not so great. JMO.
Joe DeFuria
Legend
ben6 said:
There is a BIG difference between a "benchmark fest", and an INTELLIGENT direct comparison between two technologies, of which SOME benchmarks are obviously required to observe/compare/contrast behavior.
I don't think anyone is advocating a benchmark fest.
[EDIT: ]specifically, what I'd like is relatively FEW, but relevant choices of benchmarks, that specifically contrast the behavior of the architectures in general.
We're looking for specific, quality benchmarks that help to show how the technologies are different...either performance or quality implications...or both. Again, the key is "quality" benchmarks, not "quantity". [/EDIT]
I happen to think that B3D is one of the few (only?) "Mainstream 3D Site" that is capable of doing such a comparison, otherwise, I wouldn't care if they did it or not.
I voted no.
Anytime you have a shoot you your going to have the "loser" fan base claim bais and what not. How many post have we seen that says Anadtech, Toms, HardOCP are a bunch of bias son of..... B3D does not need that. Stay in the middle stay out of shoot outs as we have plenty of other sites for that type of stuff...
Anytime you have a shoot you your going to have the "loser" fan base claim bais and what not. How many post have we seen that says Anadtech, Toms, HardOCP are a bunch of bias son of..... B3D does not need that. Stay in the middle stay out of shoot outs as we have plenty of other sites for that type of stuff...
jb said:
Yes exactly. That is why IMO "shootouts" should be extremely limited with a wide variety of cards so that you don't realy have to fill in all the time. For example if Beyond3D were to have a shoot out with only the R300 vs the NV30 today. Then a couple of weeks down the road do another R350 vs NV30 and so on it would result in what you say. But if they are done on occation with a wider variety of the most resent cards then it would be more appropriate IMO.
I think there is a definite place for shoot outs for the reasons Joe mentioned, but I think they should be restricted to interesting occasions when other sites aren't giving useful information or conclusions. For example, the nv30 versus r300 technology comparison (though not based on benchmarks) was exactly that type of technically minded shoot out, and for the reason I mention.
There is no need for the site to ever do a shoot out for the sake of doing a shoot out, but, as Tommy says, if the site wants to focus more on attracting the readership of gamers and people looking to make a final buying decision, occasional shoot-outs (and I recommend they be called something other than "reviews") certainly seem to have their place.
Also, things don't need to be "all or nothing". For example, when exploring nv30 shader performance, it might be handy to use R300 (9700, 9500, whatever) benchmarks for comparison for certain specific tests...in that case an absolute "don't compare to other architecture rule" could be a disservice, yet a dedication to such comparisons would distract from the focus of the article.
In general, I think Beyond3D should continue to be avoid shoot outs compared to how heavily other sites use them, and the criteria of technical analysis should be the focus, not whether to "shoot out" or "not to shoot out". Whether to do a shoot out or not should be evaluated after other priorites are satisfied.
There is no need for the site to ever do a shoot out for the sake of doing a shoot out, but, as Tommy says, if the site wants to focus more on attracting the readership of gamers and people looking to make a final buying decision, occasional shoot-outs (and I recommend they be called something other than "reviews") certainly seem to have their place.
Also, things don't need to be "all or nothing". For example, when exploring nv30 shader performance, it might be handy to use R300 (9700, 9500, whatever) benchmarks for comparison for certain specific tests...in that case an absolute "don't compare to other architecture rule" could be a disservice, yet a dedication to such comparisons would distract from the focus of the article.
In general, I think Beyond3D should continue to be avoid shoot outs compared to how heavily other sites use them, and the criteria of technical analysis should be the focus, not whether to "shoot out" or "not to shoot out". Whether to do a shoot out or not should be evaluated after other priorites are satisfied.
Even though it's obviously Dave's decision to ultimately make, I voted no.
I also voted no, though I would like to see comparisons between cards, or atleast be able to make comparisons.
What might be really neat would be to have a database system where you can pick a card, and see a list of drivers and cpus it's been tested with, and get scores for a set of standard benchmarks. That way, you could do things like see how scores for an individual card have changed as tests have been done with different cpus and drivers. You could also do comparisons between different cards with different drivers. Storagereview does something like this already for harddrives.
Granted, this would be some work to setup, but if Dave already does the benchmarks, it would only really require the initial database setup, and entering numbers in when benchmarks are performed. I think it could be a really nice boon for the site. Perhaps a standard set of test images could be done for reviews/previews too, which could be compared between cards to do image quality comparisons.
Anyway, just some thoughts...
Nite_Hawk
What might be really neat would be to have a database system where you can pick a card, and see a list of drivers and cpus it's been tested with, and get scores for a set of standard benchmarks. That way, you could do things like see how scores for an individual card have changed as tests have been done with different cpus and drivers. You could also do comparisons between different cards with different drivers. Storagereview does something like this already for harddrives.
Granted, this would be some work to setup, but if Dave already does the benchmarks, it would only really require the initial database setup, and entering numbers in when benchmarks are performed. I think it could be a really nice boon for the site. Perhaps a standard set of test images could be done for reviews/previews too, which could be compared between cards to do image quality comparisons.
Anyway, just some thoughts...
Nite_Hawk
horvendile
Regular
While I certainly enjoy a shootout as much as the next man, and use them for purchasing decisions, I can always use Anandtech or something similar to get my benchmark fix. But I can not use these sites to get consistently reliable investigations about the technology itself. As I'm not the first to point out, technology investigations is the strong point of Beyond3D, and I believe the site should concentrate on that. Not to the point where shootouts are dogmatically frowned upon, just so that focus is not lost.
Speaking of which: Back to studying!
Speaking of which: Back to studying!
i do not think shoot-outs are a nessesity for evey preview by any means. however i do find it disspointing to see one technlogy compared to one that is blatetnly dated and sells at a much lower pricepoint as is the case in previous previews on this site. i respect the fact that benchamrking multiple products takes time, however i think if such time is taken respect should also be given to useing that time for makeing the the most relvent i do not think shoot-outs are a necessity for every preview by any means. however i do find it disappointing to see one technology compared to one that is blatantly dated and sells at a much lower price point as is the case in previous previews on this site. i respect the fact that benchmarking multiple products takes time, however i think if such time is taken respect should also be given to using that time for making the the most relent comparisons of available products. so, not necessarily for previews, but if you are going take the time to make comparisons would be proper to make the ones most valuable to the readership. reviews on the other hand, i feel it is of the utmost importance to make such comparisons.
also i think Himself makes a great point, a standard benchmark setup would do much to make life easier for presenting such comparisons. it is not necessary for most of us to always run the absolute latest and greatest hardware and if beyond3d only upgraded their benchmarking setup every 6-12 months, as most enthusiasts do, i doubt there would be many complaints from the readership.
lastly i have to contest jb's argument. while it is true that a given fan base will be upset when their favorites are shown in a bad light, i see no reason to pander to such arrogant mentality. the fanboys can go take a piss for all i care.
so in summary, i vote yes, but the vote should be counted with consideration to my comments above.
also i think Himself makes a great point, a standard benchmark setup would do much to make life easier for presenting such comparisons. it is not necessary for most of us to always run the absolute latest and greatest hardware and if beyond3d only upgraded their benchmarking setup every 6-12 months, as most enthusiasts do, i doubt there would be many complaints from the readership.
lastly i have to contest jb's argument. while it is true that a given fan base will be upset when their favorites are shown in a bad light, i see no reason to pander to such arrogant mentality. the fanboys can go take a piss for all i care.
so in summary, i vote yes, but the vote should be counted with consideration to my comments above.
Yes.
I think some comparison (but not a mindless broadside of benchmarks) helps put the product in contemporary context, whether it's a preview or review. The (target, heh) audience here surely understands that a preview is not of a finished retail product.
And I can't always readily recall what's a "good" result in some test, so a reference to established competition would be just convenient. I'm usually too lazy to dig up earlier reviews of said competition ;-)
But the main focus should remain on what B3D does best: the dissecting and technological insight.
So to sum up my vote, I see the comparos simply as additional contextual information. Not that big a deal, but helpful, and thus it should be done, IMHO.
I think some comparison (but not a mindless broadside of benchmarks) helps put the product in contemporary context, whether it's a preview or review. The (target, heh) audience here surely understands that a preview is not of a finished retail product.
And I can't always readily recall what's a "good" result in some test, so a reference to established competition would be just convenient. I'm usually too lazy to dig up earlier reviews of said competition ;-)
But the main focus should remain on what B3D does best: the dissecting and technological insight.
So to sum up my vote, I see the comparos simply as additional contextual information. Not that big a deal, but helpful, and thus it should be done, IMHO.
- Status
Similar threads
