WTF? HardCOREware's views on cheating:

Discussion in 'Graphics and Semiconductor Industry' started by Althornin, Jun 8, 2003.

  1. Althornin

    Althornin Senior Lurker
    Veteran

    Joined:
    Feb 6, 2002
    Messages:
    1,326
    Likes Received:
    5
    http://www.hardcoreware.net/reviews/review-158-1.htm
    Let me give you a few precious pearls from this editorial:

    too bad he cant seem to do any research into the matter.
     
  2. Doomtrooper

    Veteran

    Joined:
    Feb 6, 2002
    Messages:
    3,328
    Likes Received:
    0
    Location:
    Ontario, Canada
    :lol:

    Quote:
    I'm a journalist

    A: No a guy with a webpage :!:
     
  3. WaltC

    Veteran

    Joined:
    Jul 22, 2002
    Messages:
    2,710
    Likes Received:
    8
    Location:
    BelleVue Sanatorium, Billary, NY. Patient privile
    I would add to that: "A guy with a web page who has delusions of journalism."

    --'course, now if he'd said "I'm a yellow journalist," or "I'm a tabloid journalist"....well....how could you argue?
     
  4. K.I.L.E.R

    K.I.L.E.R Retarded moron
    Veteran

    Joined:
    Jun 17, 2002
    Messages:
    2,952
    Likes Received:
    50
    Location:
    Australia, Melbourne
    Yellow journalist?

    I'm a yellow journalist (as long as that means I urinate on myself). ;) :lol:
     
  5. WaltC

    Veteran

    Joined:
    Jul 22, 2002
    Messages:
    2,710
    Likes Received:
    8
    Location:
    BelleVue Sanatorium, Billary, NY. Patient privile
    Heh-Heh...not quite what I had in mind...;)

    The term "yellow journalism" is a term for sensationalism--and a "yellow journalist" is someone who writes wildly sensational, exaggerated stories. But I think you've just put a brand-new connotation on the term which just might stick...;)
     
  6. Kristof

    Regular Alpha

    Joined:
    Jan 30, 2002
    Messages:
    733
    Likes Received:
    1
    Location:
    Abbots Langley
    Maybe we should talk about this :

    Anyone else think this is just a bad idea ? If the source is open everybody can exactly see what is done so driver writers will know exactly what to look for and tweak. Also if the full source is available you'll have tons of people playing with it and generating different versions and nobody will know what to compare with what unless you stick with one single version.

    Optimised code paths for all cards, why ? Why do people think a good benchmark would have to use different code on different hardware ? Should we re-introduce company specific APIs again so that you code to the metal of the hardware and make optimal use ? Don't think so, DX is a standard and you should use the standard and all cards should handle that standard fine with no tweaks or changes needed per card or company. Its apples versus apples, not apples versus pear, kiwi's, grapefruits, etc...

    Anyway abd idea IMHO...

    Thats because there is no such thing as FP16, FP24 and FP32... there is just FP (24+) under DX. Obviously you can PP everything but should developers really need to figure out where to put it ? When I code a shader I sure don't worry about what accuracy some operation would need to be at. But maybe thats just me.

    Err, FableMark is a stencil shadows benchmark... SIGH...

    Err, has this guy actually used 3DMark2003 since this is exactly what they offer. You can select a specific frame and render it, and even render and compare it using the ref rast. What more would you want for IQ analysis ? The filtering test also allows you to look at the filtering level boundaries. So he asks for exactly what 3DMark2003 offers, did he ever use the benchmark to its full potential ?

    IMHO he is just trying to grab some extra hits on his site...

    K~
     
  7. LeStoffer

    Veteran

    Joined:
    Feb 6, 2002
    Messages:
    1,253
    Likes Received:
    13
    Location:
    Land of the 25% VAT
    Kristof, dunno if you missed this:

    http://216.180.225.194/~beyond3d/forum/viewtopic.php?p=126626#126626

    Anyhow, the problem IMHO is that we will then primarily be benchmarking the driver team.
     
  8. WaltC

    Veteran

    Joined:
    Jul 22, 2002
    Messages:
    2,710
    Likes Received:
    8
    Location:
    BelleVue Sanatorium, Billary, NY. Patient privile
    I agree completely with this and all of your points. He seems to think the whole purpose of a benchmark is to showcase best-case hardware performance without regard to to ensuring that the processors tested are all doing the same amount of work in an environment close to what you find in shipping games. He just doesn't understand that principle or how it is germane to comparative hardware benching, apparently. The irony is that running 3DMark 03 in the 330 state is very, very much like running 98% + of all of the 3D games that ship, since the hardware companies do not optimize their drivers heavily for them and the games themselves contain no highly optimized vendor paths. The purpose of such a benchmark is to benefit the consumer--not any particular hardware vendor--and it does so by illustrating likely-case performance instead of special-case performance.

    Behind all of the assorted nonsense circulating about "what's wrong with 3DMark" you'll find nVidia, simply because the company doesn't want the average-case performance of its current hardware to become exposed in any fashion. It's understandable, but still deplorable. It's also equally understandable that some people with web pages are confused about the issue, and unfortunate. The best benchmarks primarily benefit the consumer--the worst ones benefit the hardware companies. How many consumers want a benchmark to describe product performance while running 2% of all titles shipped? Certainly, if you run nothing but that 2% then such a benchmark would have value. I'd rather look at a benchmark which describes performance for the 98% of titles shipped that do not contain optimized vendor paths, and for which the drivers have not been heavily hacked on a special-case basis.
     
  9. CorwinB

    Regular

    Joined:
    May 27, 2003
    Messages:
    274
    Likes Received:
    0
    Agreed for the second part, making it meaningful would require websites explaining which version they use. Not a big problem, but I certainly think the "guys with webpages" could use some super-dooper "tweaked" versions from their friends the IHVs...

    For the first part, recent history has proven that a closed-source benchmark was very vulnerable to "optimizations". At least, with an open-source benchmark, you can modify it at home and recompile it, which gives lots of ways of finding the various cheats^H^H^H^H^H^Hoptimizations (changing textures, shaders, camera paths...).

    What the author doesn't understand is that the only way to get total exposure would be to have an open-source benchmark and open-source drivers. The second part just isn't going to happen, and for good reasons.
     
  10. Son Goku

    Newcomer

    Joined:
    Sep 26, 2002
    Messages:
    22
    Likes Received:
    0
    What I was thinking when I read that was not the benchmark that are open source, but the drivers. In fact I was thinking it might be nice if we could see the driver code for the newer nVidia drivers to see exactly what cheats do or do not remain.

    The only way this would be exposed though is through decompilation...which is something that most web page owners probably don't know about, and also I for one wouldn't want to publish an article on "what I found decompiling their driver". Reverse engineering is something many do, but few discuss openly. In fact this is what's implied with how ATI found the lattest cheat "The ATI folks say they were studying the Detonator FX driver's new filtering routines, to see what they could learn, when they discovered this quirk...", but in tipping tech-report.com off pointed out the renaming of 3dmark03.exe...

    But open source drivers would allow one to detect who is cheating and who is not. Of course, if a company wants to cheat on a benchmark, they aren't going to want to get caught, so...
     
  11. OpenGL guy

    Veteran

    Joined:
    Feb 6, 2002
    Messages:
    2,357
    Likes Received:
    28
    You'll never see open source drivers because there is far too much IP in them. Hell, you can get a good idea of how a chip works just looking at the register specs and drivers contain a lot more than that.
     
  12. epicstruggle

    epicstruggle Passenger on Serenity
    Veteran

    Joined:
    Jul 24, 2002
    Messages:
    1,903
    Likes Received:
    45
    Location:
    Object in Space
    first open source drivers wont happen since that would release to the public many patents that are only licenced (but not owned) by nv, ati,...
    second what is the point of having a standard dx, if everyone is going to optimize their code their own way. I agree with you 100% kristof. ;)

    later,
     
  13. Bjorn

    Veteran

    Joined:
    Feb 6, 2002
    Messages:
    1,775
    Likes Received:
    1
    Location:
    Luleå, Sweden
    Doesn't the BETA members of FM already have access to the source code ?

    I don't see this as being a problem as long as you do what SA proposed. Keep one codepath fixed and let the IHV's make their own codepaths and use both while testing. And everybody would see what the IHV did to gain performance and could have them explain how that could lead to the supposed performance increase in question. Now, they could of course still cheat since the code still has to pass the drivers.

    But, if you allow the IHV's to do this you'd create a situation where the they couldn't really blame the benchmark if the performance is crappy. Thus, cheating (or should i say optimizing :)) would create a lot bigger problem for a IHV if they were caught. And the risk of getting caught is rather big since the code (i'm guessing that we're talking about the shader code only) is open source.

    There's a standard API but is there a "standard way" of using the API ?
    What we do know is that you can use the standard API differently and gain performance even while doing exactly the same thing. Thus, how are you going to create a codepath that you are sure is optimal for all IHV's ? Maybe not optimal but completely neutral ?

    The probable answer is, you can't.

    And won't this problem be bigger and bigger the more advanced the hardware will become ?

    Another thing, we're moving towards HLSL's (quickly ?) and then the "apples to apples" comparision will be a thing of the past anyway. Though hopefully only AFA the workload goes then and not the image quality.

    /Note: Opinions from a non 3D programmer :)
     
  14. Radea

    Newcomer

    Joined:
    Feb 23, 2003
    Messages:
    19
    Likes Received:
    0
    Location:
    United States
    He took ATI out of there now :?
     
Loading...

Share This Page

  • About Us

    Beyond3D has been around for over a decade and prides itself on being the best place on the web for in-depth, technically-driven discussion and analysis of 3D graphics hardware. If you love pixels and transistors, you've come to the right place!

    Beyond3D is proudly published by GPU Tools Ltd.
Loading...