The Way its Meant to be Reviewed?

Discussion in 'Beyond3D News' started by Dave Baumann, Dec 8, 2003.

  1. Dave Baumann

    Dave Baumann Gamerscore Wh...
    Moderator Legend

    Joined:
    Jan 29, 2002
    Messages:
    14,079
    Likes Received:
    648
    Location:
    O Canada!
    [​IMG]Since the release of 3DMark03 the question, largely driven by the IHV’s themselves, for 3D reviews has been to run or not to run synthetic benchmarks. While some publications have removed them and others written editorials, Beyond3D has stuck to including them – our content requires it and you, our readers, expect it. Now though we’ve come to ask the question, however rather than looking for opinions from the usual suspects, we put it to a few game developers and ask what they think is right. Here we’ve got comments from Tim Sweeney (Epic – Unreal), Markus Maki (Remedy – Max Payne), Dany Lepage (Ubisoft – Splinter Cell) and John Carmack also gives us his opinion.

    Read the full article here
     
  2. digitalwanderer

    digitalwanderer Dangerously Mirthful
    Legend

    Joined:
    Feb 19, 2002
    Messages:
    17,261
    Likes Received:
    1,778
    Location:
    Winfield, IN USA
  3. micron

    micron Diamond Viper 550
    Veteran

    Joined:
    Feb 23, 2003
    Messages:
    1,189
    Likes Received:
    12
    Location:
    U.S.
    I had to copy and paste the article to wordpad because the flashing ad's were just too much....the article was excellent though.
     
  4. Anonymous

    Veteran

    Joined:
    May 12, 1978
    Messages:
    3,263
    Likes Received:
    0
    Excellent article indeed.
    I hope this puts to rest some of the absurdist grousing about synthetic benchmarks, but I have little faith that even the game developers themselves would convince the curmudgeonly holdouts that their attitude is unscientific and counterproductive.
     
  5. digitalwanderer

    digitalwanderer Dangerously Mirthful
    Legend

    Joined:
    Feb 19, 2002
    Messages:
    17,261
    Likes Received:
    1,778
    Location:
    Winfield, IN USA
    One question sort of tugged at me while reading the article....why are you depressed about the current state of reviews? (I can guess, but I'd rather hear it if you're willing to share it.)
     
  6. Anonymous

    Veteran

    Joined:
    May 12, 1978
    Messages:
    3,263
    Likes Received:
    0
    Getting cynical in our old age, aren't we Mr. Baumann...?

    How could you expect anything different? Your perspective is one of academic investigation, but academic rigor is not what the various vendors are keen to espouse. Too much moolah involved for that. This has been commented about ad nausea in these forums...
     
  7. Tim Murray

    Tim Murray the Windom Earle of mobile SOCs
    Veteran

    Joined:
    May 25, 2003
    Messages:
    3,278
    Likes Received:
    66
    Location:
    Mountain View, CA
    first nitpick: Shadermark 2.0 isn't open-source.. or if it is, I've been sadly misled by everyone on the face of the planet
     
  8. digitalwanderer

    digitalwanderer Dangerously Mirthful
    Legend

    Joined:
    Feb 19, 2002
    Messages:
    17,261
    Likes Received:
    1,778
    Location:
    Winfield, IN USA
    Uhm, the article was written by Mr. Tan. Mr. Baumann just wrote the post here linking to it. :)
     
  9. Anonymous

    Veteran

    Joined:
    May 12, 1978
    Messages:
    3,263
    Likes Received:
    0
    well the HLSL shaders for it are
     
  10. Tim Murray

    Tim Murray the Windom Earle of mobile SOCs
    Veteran

    Joined:
    May 25, 2003
    Messages:
    3,278
    Likes Received:
    66
    Location:
    Mountain View, CA
    but Carmack's talking about The Whole Damn Thing, or so I assumed, so I would guess that for a synthetic benchmark, having an open-source engine behind it would be a definite good thing. that's why I like RightMark.
     
  11. Reverend

    Banned

    Joined:
    Jan 31, 2002
    Messages:
    3,266
    Likes Received:
    24
    What action do you think FM would take, if they were to take action?

    I believe there will not be any further patches to 3DMark03 that are meant to specifically address the issues of drivers that do not abide by FM's optimization policy guidelines.


    I gave some of the reasons in that email I sent to developers (the email is in the article, page 2). Perhaps "depressed" is not the word... maybe "frustrated". Essentially, I'm just thinking that there must be a better way of providing information in web reviews and the fact that I can't think of any (that is logical to do) makes me kinda both depressed and frustrated.
     
  12. nelg

    Veteran

    Joined:
    Jan 26, 2003
    Messages:
    1,557
    Likes Received:
    42
    Location:
    Toronto
    Good article. It was nice to hear the perspective of developers. IMHO quite frankly, I think the use of some synthetics are inappropriate for other sites. Its not just about “having the right toolsâ€￾ but “having the right tools in the right handsâ€￾. How many sites would be able to demonstrate the difference between Fp24 and Fp32 with a real world example? Most would probably just say that 32 >24 therefore better but would exclude any context as to why and at what price.
    http://www.beyond3d.com/reviews/triplex/redair9600pro/index.php?p=4#traod
     
  13. MaxPower_NVN

    Newcomer

    Joined:
    Dec 9, 2003
    Messages:
    63
    Likes Received:
    0
    Nice article. However, I disagree with the gist of the sentence above. More specifically, this portion:

    On the contrary, I think most everyone does not have to wonder about these criticisms. With everything surrounding the 3DMark2003/NVIDIA/ATI spaghetti-mess earlier this year it's obvious that the credibility, thoroughness and accuracy of both 3DMark2003 and the competing video card company's drivers were in serious question. There has been some ebb and flow since that time and so the issue has stabalized. 3DMark2003 is obviously not the only synthetic benchmark program (albeit the most popular to the layman which is why I focus on it in this reply). We saw a domino effect of sorts take place from that point on though with questions then being raised about AquaMark3, etc.

    Perhaps I take a more simpleton approach by focusing primarily on games in my reviews over at nvnews.net. I do this, though, because that is what our reader base demands. They simply do not ask for more synthetic benchmarks...they want to see games (many games) included in a video card review. They want to see screenshots comparing IQ.

    I do not discount the importance of synthetic benchmarks in reviews, they have their place. In recent months I've come around a bit from where I once stood and do give synthetics more attention. At the end of the day though, we don't buy video cards to run synthetics. Also, though some synthetics can shed some limited light on future game performance--today--they are far, far from a silver bullet (rather just some powder in the cartridge IMO). This may change and I hope it does. Open source synthetics could really help the current situation if implemented/supported correctly.
     
  14. Reverend

    Banned

    Joined:
    Jan 31, 2002
    Messages:
    3,266
    Likes Received:
    24
    Some do actually. But what you're saying is from the point of view of the buyer (and presumably a gamer). This article is from the point of view of a reviewer, of which the huge majority do not buy the video cards they review. And this article is attempting to explain why the information provided by running synthetic benchmarks may help someone (considering the purchase of a video card) in the longer run, beyond "buying cards to play games now".

    You're looking at the whole thing from a single angle and from one side of the coin. We're not suggesting that anyone that buys a video card to play games to go and run synthetic benchmarks that they don't understand, neither the meaning of the performance data of synthetic benchmarks nor what these performance data can mean in the context of how games may be developed. This is what I/Beyond3D think reviewers should be doing from the outset. If you don't think such a "responsibility" should rest on a reviewer's shoulders, I'll respect that however.
     
  15. Tim Murray

    Tim Murray the Windom Earle of mobile SOCs
    Veteran

    Joined:
    May 25, 2003
    Messages:
    3,278
    Likes Received:
    66
    Location:
    Mountain View, CA
    Synthetics are useful for judging the performance delta between two cards with regards to features that are not yet used in benchmarkable games but will presumably be present in the future. There, that's the whole damn article summed up in a sentence. ;)

    So, for a while, that was PS2.0, but the time of testing PS2.0 primarily through synthetics is either at an end or nearing an end, depending on whom you ask. But, synthetics are often only useful for determining theoretical performance in a very specific situation. For example, you don't get a realistic picture of the NV3x's shader performance from 3DMark03. You get a realistic view of its shader performance when it uses only FP32; most games won't use pure full-precision, so its performance when using only FP32 is, to a degree, moot. Meh. I don't know where I'm going, really. Synthetics are something you need lots of in order to make any sort of useful claims, and closed-source synthetics are all but worthless.

    Now I'm just playing "how many times can I edit this single post?" Rev, is the first sentence of this post basically all you were trying to say with the article? Am I in Detroit while the point is in NYC? Because... why the hell are you surprised? There's been no major architectural changes since the NV30, just refreshes, so synthetic benchmarks aren't going to test any new features--there AREN'T any more new features to test (well, PS2.0, but you have games for that now). Come NV40/R420, every single member of the "Synthetic Benchmarks Are Bad" crowd will be lining up to get their hands on a press copy of 3DMark04 or whatever the hell the synthetic benchmark of choice for PS3.0 and such will be. Why? Because they'll look like idiots if they just say, "Well, NV40 should be able to run Doom 3 at a really high resolution..." or "R420 should be able to run HL2 with antialiasing!" It's just a pendulum--once the new cards come out, synths will be all the rage, and then someone will start cheating, and then they'll be crappy, and then the cycle will repeat once NV50/R500 come out.
     
  16. MaxPower_NVN

    Newcomer

    Joined:
    Dec 9, 2003
    Messages:
    63
    Likes Received:
    0
    Not much fun in that though. ;)

    Oh I realize this and I appreciate your efforts in bringing all of this together for reviewers to chew on. What I'm trying to say is that, based on consumer feedback (they're the bottom line after all aren't they?), my attention as a reviewer is steered much more towards games and actual daily use of the hardware in question.

    I didn't mean to imply that this responsibility should not rest on a reviewer's shoulders. It should, but to a minimal degree IMO at this point in time. I'm very open to changing my personal stance on this should a better solution come about.
     
  17. nelg

    Veteran

    Joined:
    Jan 26, 2003
    Messages:
    1,557
    Likes Received:
    42
    Location:
    Toronto
    I wonder just how much of this interest stems from people trying to justify the purchase that they already made.
     
  18. MaxPower_NVN

    Newcomer

    Joined:
    Dec 9, 2003
    Messages:
    63
    Likes Received:
    0
    Yes, that is a valid question. But they could just as easily try to justify their purchase with synthetics...yet they opt not to do so. Why? My opinion is that synthetics just don't compare to games regarding credibility. Neither of them (games or synthetics) are perfect measurements but games are vastly the lesser of the two evils as it were. ;)

    Another thought, how much of this interest stems from people simply wanting to know how well card "X" will run any number of games currently available? My opinion (based on very loose stats :)) is that the majority of this interest is for this reason and not the one you suggest.
     
  19. Althornin

    Althornin Senior Lurker
    Veteran

    Joined:
    Feb 6, 2002
    Messages:
    1,326
    Likes Received:
    5
    why are games "the lesser of two evils"??
    Time and again this year the synthetics have been shown to be vastly more accurate in predicting future gaming ability than games. I dont see why you have this incredibly anti-synthetic view.

    As per your last sentence, i think you are wrong. People dont buy cards for 3 month usage. They buy for a year to 18 months, and that means future games that use features current games do not. Most users are uninformed, and think that todays games will accurately reflect relative performance in tomorrows games. In this, they are wrong, and it is your job as reviewer to fill them in on the truth - not to bow before their lack of knowledge.
     
  20. Pete

    Pete Moderate Nuisance
    Moderator Veteran

    Joined:
    Feb 7, 2002
    Messages:
    4,943
    Likes Received:
    347
    Nice article, Rev. Two things struck me as I read it. After a few bandages and muttered insults, I'm able to discuss them:

    1. You've got a monster of a run-on sentence on the first page. I guess B3D's editors are on hiatus? ;) </nitpick>

    2. The idea of buying a card to last 18 mo.'s is, I think, limited to the high-end and perhaps mainstream Most people still buy budget cards, however, and I have a hard time thinking a 5200 will last 18-24mo's with decent framerates. Maybe I'm being overly finicky, though. This is probably related more to other sites, though, as B3D is more about hardware analysis, and budget cards tend to be retreads of last year's tech. Still, the FX 5200 makes my observation more relevent than in the past (GF4MX ~= GF2, R9000 ~= R8500).

    Speaking of synthetics, I'm more than a little curious why only Lars noted in an official article that nV doubled performance in AM3 from Det44 to Det 52. Did I miss out on a related B3D thread that got to the bottom of this? Have people asked Massive why this happened? I have a hard time believing nV's drivers were so unoptimized.

    Speaking of predicting performance, I'd also be interested in an article comparing performance boosts from a card's first released drivers to the latest ones (i.e., a 9700P with the first Cat's to the current 3.9's). It'd be hugely time-consuming, but it'd also be potentially hugely informative as to a manufacturer's track record WRT initial and optimal performance. As Tim Sweeney said, though, the generational leaps from DX6 to DX9 were great--possibly so great as to render (har-har) driver improvement comparisons across generations invalid, or at least only minimally useful as a predictor of future card + driver interactions.
     
Loading...

Share This Page

  • About Us

    Beyond3D has been around for over a decade and prides itself on being the best place on the web for in-depth, technically-driven discussion and analysis of 3D graphics hardware. If you love pixels and transistors, you've come to the right place!

    Beyond3D is proudly published by GPU Tools Ltd.
Loading...