Motherboards.org uses 53.03 benching 3dm2k3?!?

Discussion in 'Graphics and Semiconductor Industry' started by digitalwanderer, Jan 18, 2004.

  1. digitalwanderer

    digitalwanderer Dangerously Mirthful
    Legend

    Joined:
    Feb 19, 2002
    Messages:
    17,276
    Likes Received:
    1,788
    Location:
    Winfield, IN USA
    Uhm, ByteMe? We're all trying rather hard to keep this civil and not get the thread closed, could you try and not outright insult Brent next time? (Brent really is alright, even though I don't agree with him on an awful lot of things. :) )

    EDITED BITS: Besides, Brent ain't ignorant. I disagree with him, but I do respect him for being one of the brighter bulbs on the x-mas tree.
     
  2. illuminati

    Newcomer

    Joined:
    Jan 21, 2004
    Messages:
    15
    Likes Received:
    0
    Hey all,

    This is my first post here, and definitely won't be my last. Looks like a great group of members here!

    I just wanted to throw out my "open" view...

    First off... Worm did contact me and inform me of the unapproved driver I used in my 9600XT review. I have since removed the 3DMark03 results to avoid unreliable tests. I have thanked him for bringing this to my attention. Before yesterday, I was unaware that FM had an "Approved Driver List" for use with 3DMark03.

    Now on to my view: I see where some people come from when they say that synthetic benchmarks do not show much value when evaluating a card because I want to see which card performs the best in "real" situations. Like Brent mentioned, just because the synthetic benchmark shows the true potential of a card, does not mean that will be the actual performance noticed in actual games.

    I also think there is a difference between optimizations for synthetic tests and optimizations for real games. Game performance is very important and I think that game makers should have the right to make their game perform as best as possible in order to give their customers the best performance their hardware can provide. I however do not think that optimizations for synthetic tests are good because that does sway test results and is not meet the purpose of synthetic tests. As stated earlier, synthetic tests should be a good way to show the future power of your hardware. It is sad that optimizations are used to diminish that goal.

    I also like seeing tests that utilize a hardware manufacturer's most recent driver because I feel it is important to see how those drivers perform. However, it seems as though the only way to effectively test the most recent driver is to use real games in order to rule out the possiblity that there are optimizations for synthetic tests included in that driver. And I think it is safe to ignore optimizations related to games because that is actually performance gained to the user for that card.

    Bottom Line: Optimizations related to synthetic benchmarks are shady because the user does not necessarily gain that performance, but optimizations in games actually increase the playability and the user's performance with their specific hardware.

    Please let me know if this viewpoint makes sense. Like I said, this is an "open" view point, and I'm currently doing more research to better understand this whole issue.
     
  3. ByteMe

    Banned

    Joined:
    May 27, 2003
    Messages:
    136
    Likes Received:
    7

    What I said and is now deleted was not an "insult". I in now way was attacking him. What I said was the truth. Part of the problem is (here included) that people are so worried about being PC that the crap that happens (i.i. FM / Nvidia) gets shoved under the table. I could not say very much nice about [H] without crossing my fingers.

    It dissappoints me greatly that people aren't screaming murder. Instead they take it like sheep. And this is "the greatest" 3 D related site. I think I need to go take a shower.
     
  4. digitalwanderer

    digitalwanderer Dangerously Mirthful
    Legend

    Joined:
    Feb 19, 2002
    Messages:
    17,276
    Likes Received:
    1,788
    Location:
    Winfield, IN USA
    I can't argue with personal hygiene, but you did directly insult Brent in your previous post when ya called him a jerk and he really is anything but.

    Brent comes here and talks about his views and why he has them, without saying we're all full of crap for not agreeing with him; I really do respect the hell out of him for that.

    I don't agree with him a lot, I sometimes think he answers things the way he does because of his job, but I think he is a good person who respects others views and bends over backwards to help people out and share his knowledge as well as his ignorance....I can't say that about the majority of people I see posting on boards.

    You can be insulting without insulting someone, that's allowed. It takes a bit of extry clever, but you've got plenty of that in that arena. ;)

    Mebbe it is PC, I prefer to think of it as common curteousy and manners. It's something I like about B3D, it's a nice change of pace from most places in that respect....they make me toe the line pretty well. :)

    EDITED BITS: One quick thing I forgot: no one is asking you to say anything nice about [H], they're asking you to not directly insult someone who works there and who has proved to the community here that just because he works at [H] it doesn't mean that he's an asshat who thinks he knows more than he does. (He ain't, Brent really is one of the good guys damn it!)
     
  5. digitalwanderer

    digitalwanderer Dangerously Mirthful
    Legend

    Joined:
    Feb 19, 2002
    Messages:
    17,276
    Likes Received:
    1,788
    Location:
    Winfield, IN USA
    I think you did miss one, but I can't remember it either. (Was Max Payne2 dx9 or pseudo-dx9?)

    I'm glad you posted that, I was sort of afraid I was the only one who thought that all the "big dx9 titles" have pretty much sucked so far. :)
     
  6. illuminati

    Newcomer

    Joined:
    Jan 21, 2004
    Messages:
    15
    Likes Received:
    0
    I'm actually enjoying Halo now, but that could be because I never played it on XBox. :) I however did not like TRAOD at all.
     
  7. Tim Murray

    Tim Murray the Windom Earle of mobile SOCs
    Veteran

    Joined:
    May 25, 2003
    Messages:
    3,278
    Likes Received:
    66
    Location:
    Mountain View, CA
    Lessee... first of all, I thought Tron 2.0 was DX8, maybe 8.1. Not entirely sure--didn't really like the game either way.

    SW: Galaxies uses PS2.0 in the water. Not a bad visual quality increase, either.

    Now...

    Nobody disagrees with you there. At least, anyone who doesn't believe that Futuremark is the devil for not including partial precision flags in 3DMark03 GT4 and that NVIDIA is in the right for "optimizing" for it (even though the optimizations are more than just reduced precision in GT4--they optimize GT2 and 3, but you can find all of that elsewhere).

    This statement is troublesome. On the one hand, you have synthetic gaming benchmarks like 3DMark's game tests. On the other, you have purely synthetic benchmarks which, if they are good, merely stress the hell out of a very specific feature on a video card. For example, with Shadermark, you can test the partial-precision PS2.0 capabilities of a card and compare that to full-precision results to see what kind of improvement using _pp brings in a card.

    As a measure of future game performance, the first is all but useless. It is, at best, a conjecture of which direction the game industry will take, what features will be used, what features will not be used, etc. You can't take a 3DMark GT4 result and translate it into HL2 or D3 or Halo 2 performance.

    However, pure synthetic benchmarks' results mean nothing in a vacuum. They can, however, be applied by an individual to form a conjecture on how an card will perform in a certain game. For example, when HL2 was first announced, Valve said, "It uses a lot a lot a lot of PS2.0 shaders." I don't think anyone who had seen the purely synthetic benchmark results from ATI and NVIDIA cards were surprised at all at the margin ATI was in front of NVIDIA. In fact, the delta between the two in HL2 is almost the same as the delta between them in almost every other PS2.0 benchmark out there (or so I"ve heard, repeatedly, from a number of people, so I don't really think that's incorrect. I'm just kinda too lazy to check the numbers myself at the moment). So once you know what features a game is dependent upon, you can begin to estimate how a certain card will perform based upon synthetic benchmarks. THAT'S why they're useful, not as an all-inclusive estimate of future game performance. Doom 3 is not HL2 is not Far Cry is not whatever floats your boat--no single benchmark can approximate performance in all of those games. However, a single benchmark can approximate performance of a heavily used feature in one of those games, so if you believe that feature to be the bottleneck.. yeah, rambling again, and you get the point. Shutting up now.

    Slap me, Johnny, I'm on my crazy synthetic benchmark rant that makes no sense.
     
  8. nelg

    Veteran

    Joined:
    Jan 26, 2003
    Messages:
    1,557
    Likes Received:
    42
    Location:
    Toronto
    Welcome! As to not rehash the 3Dmark/synthetic issue over again (there are many threads here that leave no stone unturned), perhaps the best endorsement for their use would be the fact that Beyond3D see’s the purpose they serve.
     
  9. Althornin

    Althornin Senior Lurker
    Veteran

    Joined:
    Feb 6, 2002
    Messages:
    1,326
    Likes Received:
    5
    This is where we disagree. REPRESENT !=exact correlation. They give you an IDEA of future performance - meaning people who used synthetics KNEW that in TR:AOD you'd get better performance on ATI cards 6 months before anyone else did. Not showing that view does a disservice to your readers - especially to gamers, who want to know what thier cards will do in games in the future.
     
  10. illuminati

    Newcomer

    Joined:
    Jan 21, 2004
    Messages:
    15
    Likes Received:
    0
    Thanks for the feedback Tim! I do see the value of synthetic benchmarks and hope synthetic benchmarks soon have a more positive rep. :shock:
     
  11. Althornin

    Althornin Senior Lurker
    Veteran

    Joined:
    Feb 6, 2002
    Messages:
    1,326
    Likes Received:
    5
    Do you benchmark EVERY game?
    No? Then you are taking a "slice" of the gaming pie, per se, and showing it as representative of the cards general behavior in todays games. If the card is optimized in most popular benchmarks, but not in most games, then you are showing a biased and untrue representation of general performance.
    End of story.
     
  12. kyleb

    Veteran

    Joined:
    Nov 21, 2002
    Messages:
    4,165
    Likes Received:
    52

    well i don't belive futuremark is wrong for not includeing pp-hints and i don't think it is right for nvidia to take it upon themselves to reduce precision. however, real games can be specificly targeted for "optimsations" in such a way that the peformace increase only effects the benchmarks and does not transfer to increased ingame peformace. when looking at the defualt timedemos in ut2003 in compasion to custom ones, it seems pretty clear that nvidia is doing exactly this.
     
  13. demalion

    Veteran

    Joined:
    Feb 7, 2002
    Messages:
    2,024
    Likes Received:
    1
    Location:
    CT
    Revisiting the synthetics discussion

    But it does mean which hardware is faster at a particular workload. A good synthetic test gives you detailed information on what the workloads are.

    Replicating that level of specific information and isolation is indeed just as possible for a game, if you go out and do the work and analysis to get the same information synthetics give you. Are you going to be doing this?

    They use, as some examples, optimizations that do the same workload differently (like different AA and AF methods can be done by hardware), and optimizations that replace a given workload with something else to misrepresent fps (like "on the rail" clipping plane hand tuning for time demos).

    Valid and invalid are determined by how these optimizations are represented, and what they deliver to the user. The latter, for example, is represented as "in game" experience, but actually only delivers faster timedemos/fixed playback.

    What your discussion actually confuses me about is whether you are reviewing hardware or games, and I'll explain.

    As far as the game is using the same thing the "synthetic" tests, yes it does. That's why the information it works to give you puts synthetics at and advantage.
    Let's discuss your hypothetical situation.

    Yes it does.

    What is the relevance of specifying minimum FPS? Minimum FPS is representative of a specific game experience on a system at one point in time..what about the other points in time? You're introducing that in an arbitrary context.
    A synthetic doesn't begin to pretend to define what the minimum fps will be in a given game, so why evaluate it based on that? There is no "minimum FPS for games" synthetic, is there? I'm aware of pixel shader, fillrate, vertex shader, CPU, and AGP bus synthetics though...they apply to minimum fps when the associate factor(s) does.

    What it sounds like (assuming the 4 fps difference is proportionately insignifcant, occurs at similar times, etc) is that either the system (CPU, AGP, hard disk, etc), or some other GPU factor besides PS 2.0 shading, is the determing factor when this hypothetical minimum occurs.
    How does that make the times when the fps difference is more than "just" 4 fps any less significant?

    Synthetics don't claim to represent every second of performance, so why do you hypothetically test them by that criteria and dismiss them entirely with regard to "game experience" when they fail it?

    What "games", doing what? Which "synthetic tests"? I guarantee you that there are situations where "the percentage of the difference in the FPS in the game a user will play is less than the percentatge of the difference in <any game chosen instead of a synthetic>", such that choosing a game is at least as much of a failure of your criteria as a synthetic.

    Except...that a good synthetic will set out to inform of you of when this discrepancy will happen and why, and how this applies to the hardware. Also, if you don't dismiss it out of hand, it will tell you when what it measures will actually be indiciative of gaming performance.

    It seems clear that a synthetic benchmark will be indicative when similar things are the bottleneck. I.e., a DX 9 benchmark should be indicative of the features unique to DX 9, like pixel shader and vertex shader 2.0 models at the current time.

    3dmark 03, as one example, does demonstrate success in this, when it is actually allowed to use (as one example) its DX 9 workload, and when compared to games stressing (in correlation) DX 9 workloads. Not in predicting fps, but in indicating specifically how hardware compares in executing such workloads.

    The problem that can lead to error here is when you fail to be informed of when either of these parties isn't doing what you think it does, such as DX9 features not actually being stressed.

    But this applies to both "synthetics" and "'real' games". But it is with synthetics that you are likely to have effort being put forth by the authors on correcting that error, because such error defeats their purpose...this is not true with games.

    The distinction here is not important for reviewing games, but it is crucial for reviewing hardware and the drivers that expose its features and performance.

    Now, you could indeed try and duplicate all the detail of information of synthetics with a game, and isolate the specific aspects of performance involved.

    Are you going to do this anew for every game? Are game authors going to always set aside time to help you with extracting information? If they do, it just became a synthetic benchmark...it is that isolation that is the distinction.

    When comparing PS 2.0 shaders being executed, and equivalent features being offered, yes.

    Are you still just comparing the minimum fps?
    Are you comparing PS 2.0 shaders?

    Sure, but why do you ignore the times it does parallel? That is why synthetics are not proposed by anyone (well, that I've encounteered recently, AFAIK) to replace measuring game performance, but to be an information source used to get more information from game performance measurements.

    For example, in your hypothetical situation, it might be telling you other things: pixel shader 2.0 is not the limiting factor in the game in that situation; that one of the cards might not be doing the same pixel shader workload; that one of the cards might be removing other limiting factors and you don't know about it. Synthetic pixel shader 2.0 (and vertex shader 2.0, and CPU speed, and bandwidth, and fillrate...etc) tests will help you find out which is the case, or if something else might be occuring.

    However, as long as you ensure that you are comparing the same workloads in games (harder when you throw out an information source, but you can depend on other sites to investigate for you and report on it) and don't ever offer an opinion on the information you ignored and propose it was an informed opinion, you can indeed do "this is how this hardware runs these games and these games alone" reviews accurately, because that is the effort you extended. Perhaps even excellent ones, if you take steps to ensure that something like the timedemo issue is avoided (as I understand you are).

    Do you call them gameplay experience reviews or hardware reviews? Do you still compare different hardware and draw conclusions?

    For example, I wouldn't be surprised if a 9600 non Pro was around the same speed running PS 2.0 in Halo as a 8500 when running PS 1.4, perhaps even slower, and there should be quite a few areas where the image quality will be, subjectively, very close. This comparison is indeed a valid gameplay exprience comparison, but it is uniquely uninformative about the actual differences in the hardware involved.

    Sounds like a perfect place to use a good synthetic, which has these advantages over a game:
    • It measures and isolates a variety of things.
    • t clearly informs what it is measuring.
    • It has a vested interest in successfully measuring what it says it tries to measure (to be a good synthetic).
    It shares this disadvantage:

    Any IHV can target it to misrepresent their hardware in comparison to other hardware.

    If you care about providing accurate information, it seems to me that it is always a time and place to utilizes those advantages, because it helps in overcoming the common disadvantage games and synthetics share. If that's all you use a synthetic benchmark for, it has served its purpose. That's all any "synthetics advocate" has claimed AFAICS, including Futuremark. It is simply a matter of using them in a way that makes sense...as the minimum fps comparison does not, for example.

    Out of curiosity: are you going to talk to the programmers for every game introduced and get the same level of information and detail "synthetics" will give you up front? This is actually possible with some game authors, and that would indeed serve well, as long as it happens.

    OK. But that's philosophy. It is the application of it to providing information to readers that this discussion is about.

    Then HardOCP is a place more easily exploited to misinform about hardware, and a place failing to use all available information and tools to independently ensure that it is not doing so, and educate its readers accordingly. Unless you avoid comparing hardware based on what you are willing to do, and warn users not to do the same.

    You are choosing arbitrarily more guesswork in place of information available to you, and against your stated philosophy. You will depend on the good will of IHVs to honestly present their products to you in comparison to their competion (or on their incompetence in misrepresentation), and the work of other people's investigation to correct misinformation.
    Actually, the latter should work if you honestly go back and correct any oversights, but that leaves you your readers with a lack of timely veracity. You'd have to bear the possibility of looking often wrong if IHV(s) decide to repeatedly take advantage of your popularity while you follow such a practice of giving them a smaller set of information necessary to distort.

    Also, there is doing your best to be accurate, and there is specifically choosing to give up on the endeavor in one or more areas.

    Whether one results in "correct or incorrect" information depends on whether happenstance or some party with a vested interest is working against accurate information being represented, and how well the inaccurate factor is hidden.
    Whether one is "wrong or right" depends on whether you tell the truth about the information you did and did not have available when you made your choice in investigation, and what you propose your investigation represents.

    Either way, there is "wrong and right" involved.
     
  14. cthellis42

    cthellis42 Hoopy Frood
    Legend

    Joined:
    Jun 15, 2003
    Messages:
    5,890
    Likes Received:
    33
    Location:
    Out of my gourd
    Considering PP-hints didn't exist for many moons after 3DMark03 came out, and adding it to the scored tests would destroy any semblance of score-tracking with previous results... yeah, I find it hard to see that as a sticking point. They could toss in a full-precision/partial-precision comparison test as a feature test I suppose, but one can already get that from other synthetics, and I'm not sure just what it would show. (As it's not just going PP, but also running different shader instruction orders and customizing them for a particular IHV that makes a real difference, which is in many ways what nVidia has been doing and what mass developers will NOT necessarily be doing--and certainly not what an broad-based API benchmark is supposed to show.)
     
  15. Martin Eddy

    Regular

    Joined:
    Oct 5, 2003
    Messages:
    491
    Likes Received:
    4
    Location:
    Australia,Brisbane
    :D Here Here DEMALION! :D
     
  16. John Reynolds

    John Reynolds Ecce homo
    Veteran

    Joined:
    Feb 7, 2002
    Messages:
    4,490
    Likes Received:
    262
    Location:
    Westeros
    I'd slap you for not liking Tron 2. . .that was one of my favorites last year.
     
  17. Hanners

    Regular

    Joined:
    Jul 12, 2002
    Messages:
    816
    Likes Received:
    57
    Location:
    England
    I finally got around to picking it up, just in time for ATi drivers to break it completely. :roll:

    I could play it using my 5900, but there's a lot to be said for laziness. :p
     
  18. ByteMe

    Banned

    Joined:
    May 27, 2003
    Messages:
    136
    Likes Received:
    7

    I agree with you in principle. But the problem is the scope. 3Dmark at one time was considered by many "the benchmark". So the importance of the disagreement is larger.

    If the importance of a disagreement had a value of $.02 then I could care very little and might not say very much. But since the importance of this benchmark was huge I believe that people should be much more upset at (Nvidia/[H]/FutureMark/others).

    The entire "clusterfuck" of this FM deal amazes me.





    WOOOT! NOT ONE SINGLE INSULT, so byteme! :p
     
  19. Khedlar

    Newcomer

    Joined:
    Jan 21, 2004
    Messages:
    11
    Likes Received:
    0
    First post, and I must say these is about the most knowlegdable sites I have ever seen.

    I think synthetic benchmarks have a place in any review. At the same time they should have a disclaimer on them stating that this benchmark may not represent actual performance in any specific game. I say this based on the game I play today, Horizons. ATI owners are having all kinds of problems. If I would bought a card in the last 6-9 months I would of bought a highend ATI.

    Now this may be obvious to you or I, but to the average consumer it is not. They look at the highest number and that is the card they buy only to be dissapointed when they hook it up and it dosen't perform according to the benchmark results.

    This thing between Nvidia and Futuremark is frustrating. I understand the problems. But in the end I will always use the latest drivers because they tend to fix problems with real games. If I see a review for a new product that is using older WHQL drivers I tend to dismiss it. Hell, MS provides the latest drivers through windows update.

    I understand Nvidia's posistion and Futuremarks. Until they both sit down to the table and work something out I'm affraid that 3DMark03 will continue to lose favor with reviewers. Right now both are acting like little kids, both saying they are right. No one wins and in the end it is the consumer that takes it in the shorts.
     
  20. Tim Murray

    Tim Murray the Windom Earle of mobile SOCs
    Veteran

    Joined:
    May 25, 2003
    Messages:
    3,278
    Likes Received:
    66
    Location:
    Mountain View, CA
    Just didn't do anything for me. Admittedly, I didn't play multi, but the single player felt boring, repetitive, and generally not that great. Plus the light cycle races... eeeeh.
     
Loading...

Share This Page

  • About Us

    Beyond3D has been around for over a decade and prides itself on being the best place on the web for in-depth, technically-driven discussion and analysis of 3D graphics hardware. If you love pixels and transistors, you've come to the right place!

    Beyond3D is proudly published by GPU Tools Ltd.
Loading...