FutureMark will not be accepting NON "WHQL" driver
- Thread starter T2k
- Start date
I forgot to mention the original post over @Rage3D:
Hey ATi FanATIc's, I think your going to like this!!
It looks like that FutureMark will no longer be accepting NON "WHQL/OFFICIAL" equipped systems to have their scores to be published online!! This means that the "OPTIMIZED" driver set's from Nvidia (42.67 & 42.68 & 42.69 and possible future release's) will be not valid from this day forward. ATi fans need not to worry. Since the Catalyst program has come along, driver releases have been official and fully certified. Terry from ATI has commented on this, which i posted at the end of this post.
FutureMark:
Rage3D has contacted Terry Makedon (ATI's Software Product Manager) to comment on the situation. His response is...
ATi:
EDIT: http://www.rage3d.com/#1048537578
Hey ATi FanATIc's, I think your going to like this!!
It looks like that FutureMark will no longer be accepting NON "WHQL/OFFICIAL" equipped systems to have their scores to be published online!! This means that the "OPTIMIZED" driver set's from Nvidia (42.67 & 42.68 & 42.69 and possible future release's) will be not valid from this day forward. ATi fans need not to worry. Since the Catalyst program has come along, driver releases have been official and fully certified. Terry from ATI has commented on this, which i posted at the end of this post.
FutureMark:
Rage3D has contacted Terry Makedon (ATI's Software Product Manager) to comment on the situation. His response is...
ATi:
EDIT: http://www.rage3d.com/#1048537578
That is NOT a statement saying they only accept results based on VHQL drivers. What they are saying is that they feel that nVidias 3DMark03 optimized drivers are not rendering in a representative way.
The VHQL noises is stuck on top of that.
On the OpenGL programmers forum, an nVidia employee stated that 3DMark03 was a bad benchmark because of its very inefficient programming, and that in optimising for it you could simply replace parts with much more efficient rendering special cased in the drivers. Spending time of such work felt like a waste of resources though since it doesn't transfer into a benefit for any actual application.
It may be that he was talking about the drivers in question. If so, it is no wonder that FutureMark don't want to publish the results. However, if that is the case, the drivers achieve their results by exploiting a general weakness in 3DMark that FutureMark can do nothing about - special casing in the drivers. So instead of specifying exectly why those drivers are "bad", which would be easy if for instance 16-bit FP was all it was about, they divert our attention a bit.
As far as I know, there is nothing that stops a VHQL driver from containing special case paths for particular applications. VHQL doesn't seem to fit into the equation at all, rather than sounding suitably officious.
Of course, there is nothing stopping SIS, S3, ATI from doing just what nVidia (seemingly) did. As long as they keep their mouths shut.
I have to say, that particular critisism of 3DMark is quite valid.
Entropy
Edited for removal of redundant personal opinions.
The VHQL noises is stuck on top of that.
On the OpenGL programmers forum, an nVidia employee stated that 3DMark03 was a bad benchmark because of its very inefficient programming, and that in optimising for it you could simply replace parts with much more efficient rendering special cased in the drivers. Spending time of such work felt like a waste of resources though since it doesn't transfer into a benefit for any actual application.
It may be that he was talking about the drivers in question. If so, it is no wonder that FutureMark don't want to publish the results. However, if that is the case, the drivers achieve their results by exploiting a general weakness in 3DMark that FutureMark can do nothing about - special casing in the drivers. So instead of specifying exectly why those drivers are "bad", which would be easy if for instance 16-bit FP was all it was about, they divert our attention a bit.
As far as I know, there is nothing that stops a VHQL driver from containing special case paths for particular applications. VHQL doesn't seem to fit into the equation at all, rather than sounding suitably officious.
Of course, there is nothing stopping SIS, S3, ATI from doing just what nVidia (seemingly) did. As long as they keep their mouths shut.
I have to say, that particular critisism of 3DMark is quite valid.
Entropy
Edited for removal of redundant personal opinions.
Brent said:
Hmm...well, it was in the license agreement, and they pointed that out in their rebuttal to nvidia's provided rationale for criticizing 3dmark03.
I really think your outlook in this matter is because you looked at it backwards from the beginning. You and Kyle were aware of the issues with nvidia's drivers, but the stance you took in your "good benchmarking" article was based on the issue being a fault of Futuremark, and not nvidia.
I won't argue that they "could" have made that stance a bit clearer, but in the role of objective journalist you "could" have presented what they did state more prominently than you did. What is your take on what you should have done differently in regards to this issue of WHQL driver certification?
Sorry if it re-starts an old argument, but I think it is very pertinent to your comment.
Entropy,
I think the issue of WHQL certification is to ensure that everyone is at least using FP24 (which means FP32 for GFFX) as opposed to lower, higher performance modes. WHQL certification should ensure this happens.
I think the issue of WHQL certification is to ensure that everyone is at least using FP24 (which means FP32 for GFFX) as opposed to lower, higher performance modes. WHQL certification should ensure this happens.
demalion said:
The problem is that there are demonstrably effective methods of increasing 3DMark scores by special casing in the drivers, and that there is no way for us to detect it if done. Yes, I'd say that this was a problem, for anyone trying to interpret the results.
(Just as the lack of "correct output" requirements are a problem. That's not to say that FutureMark can easily solve these problems. As far as correct output goes they have at least made comparisons easier now. We have already seen those features put to good use on the web.)
Now this is a competitive market - draw your own conclusions as to what IHVs will do in order to help shift their product. Your guess is as good as mine - and that's the problem.
I have no interest whatsoever in pinning blame or guilt on anyone - I'm neither a judge or a priest. People should simply be aware, as far as I'm concerned.
Entropy
Hmm...this is a demonstrated problem with anything recognized as a benchmark. It is a fault, but it seems a confusion to mention only that it as a fault of the the application (which is the exact phrasing you used), since the fault is universal among applications used in that fashion. The only specific entity that it seems reasonable to associate the fault with is the party performing the act of cheating.
Therefore, I object to this:
where I think "nvidia's 3dmark specific drivers" are the issue, but I do not object to this
I just happen to think that your first statement did not follow your stated intent in the second one.
Further details on my viewpoint (perhaps a bit dated?) can be found here.
And, for contrast, my view on some of their "zixel" performance figures as I understand them can be found here.
Therefore, I object to this:
where I think "nvidia's 3dmark specific drivers" are the issue, but I do not object to this
I just happen to think that your first statement did not follow your stated intent in the second one.
Further details on my viewpoint (perhaps a bit dated?) can be found here.
And, for contrast, my view on some of their "zixel" performance figures as I understand them can be found here.
Since it's you demalion, I'll give you a long response.demalion said:
Well, while the mechanism could be used on any benchmark, it is arguably more of a problem for 3DMark for two reasons.
First, less important, 3DMark is synthetic - there are no incidental performance benefits for any actual users per se.
Second, and most important, 3DMark has such a high profile and such longevity as a benchmarking standard, that an effort to squeeze as high scores out of it as possible is sure to pay off in the market. Both in terms of product/brand recognition and in revenue.
Neither of the above can be said to be a fault of Futuremark - it is hardly wrong of them to produce a benchmark with outstanding recognition and product revue penetration. On the other hand, this fact also makes 3DMark particularly profitable to, let us say, "spend effort optimizing for". Any such application specific optimization will reduce the predictive value of the results. Application specific code only makes the problem worse.
So it isn't strange that FutureMark has dealt with these things a bit furtively. If they point the finger at a sinner they are also calling attention to a huge problem with the predictive value of their benchmark. They're in a bit of a bind, quite simply.
(You may remember the hubbub that resulted when it was discovered that disabling the splash screen between tests caused nVidia scores to drop? FutureMark already knew about this and even had a note about it on their site where they just said that it had nothing to do with their code and that it was due to nVidias drivers. But they called no attention to it, didn't say what was going on, just made sure they washed their hands of it. Nor have I ever seen a word about the horrid image quality "optimizations" of the Xabres.)
I'm arguably more interested in benchmarking than 3D - I've been taking an active interest since the formation of SPEC. Before that, actually. And if there is one thing you can be sure of, it is that IHVs will do everything they can to make their product look good by whatever yardstick is used by their market. The approach taken by nVidia vis a vis 3DMark is exactly the same that for instance SUN took vs SPEC and later TPC. If you can't compete, discredit and direct the attention towards something else. And sometimes this is actually valid - benchmarks do loose usefulness, and the industry should move on to something that better reflects where the focus lies.
One of the valid criticisms levied against SPEC is that it is as much of a compiler benchmark as a hardware benchmark. The flip side of that coin is that there is probably nothing that has been as important for pushing the quality of compiler generated code forward as the existance of that well defined and widely acknowledged base of target codes. I've always seen the same as potentially true for 3DMark - that the existance of a widely used benchmark application could help push driver performance generally. Of course, this would still make it less useful as a predictor, but the incidental benefit would far outweigh that the masses are conned.
Note however that application specific code completely invalidates this potentially beneficial aspect of 3DMark.
Entropy
PS: Carmacks engines have been targets in the same way on the OpenGL side. I haven't heard of anyone being able to bypass his routines and replace them with equivalent but faster though. Avoiding inefficiencies or worse, redundancies is important in benchmarks. For instance once compilers got smarter they simply optimised away entire inner loops of some older widely used benchmarks because they figured out that the results weren't used globally.
Also Carmack's coce is actually used in games which is obvious, so it helps everyone who uses iD based games if companies optimize for it.
That was very well worded entropy, I must say everything made perfect sense and not really anything was arguable. Just have to say I am impressed when people do such a good juob at addressing something and in a totally unangry way. (well being coherent is usually helped by not being irate)
That was very well worded entropy, I must say everything made perfect sense and not really anything was arguable. Just have to say I am impressed when people do such a good juob at addressing something and in a totally unangry way. (well being coherent is usually helped by not being irate)
Rage3D seems to breed the worst kind of videocard trolls around these days.
NV30 is already kicking ass in 3dmark2003 with new drivers, as the long xbitlabs(?) comparison shows, and it was even doing substantially better than r300 a few months back on the doom3 alpha.
People keep dissing nvidia with their problems over getting nv30 to market, but with a few driver revisions and det 50 around the corner - all the signs point to nv30 delivering on its promises.
NV30 is already kicking ass in 3dmark2003 with new drivers, as the long xbitlabs(?) comparison shows, and it was even doing substantially better than r300 a few months back on the doom3 alpha.
People keep dissing nvidia with their problems over getting nv30 to market, but with a few driver revisions and det 50 around the corner - all the signs point to nv30 delivering on its promises.
bridpop said:
Arent those the same drivers that have precision issues that forced Futuremark to remove them?
And i havent read the comparison, but what level of AA/aniso? And was the slider set to "Application" or Balanced?
That last is the primary question....
All the signs point to nv30 NOT delivering its promises, IMO.
As does Beyond 3D, apparently.bridpop said:
Poppycock. What Carmack said what that the NV30 was half the speed of the R300 in the ARB2 (i.e. highest precision) path but that the NV30 was slightly faster than the R300 when it used the NV30 path, which has slightly worse quality than the ARB2 path.
Any time you want quality, you take a (massive) performance hit. If that's cinematic rendering, then I guess nvidia is "delivering" all right.
And where are the WHQL drivers for the NV30? More of nvidia "delivering", eh?
-FUDie
I noticed the xbitlabs article a while ago.
Name the worst assumptions duped reviewers make concerning GF FX drivers, they made them.
They used default settings (so their trilinear benchmarks are not really trilinear).
They used the 3dmark03 optimized drivers. They did do a rather unique image quality check, though...to quote them:
I guess it is better than being completely blind.
Name the worst assumptions duped reviewers make concerning GF FX drivers, they made them.
They used default settings (so their trilinear benchmarks are not really trilinear).
They used the 3dmark03 optimized drivers. They did do a rather unique image quality check, though...to quote them:
I guess it is better than being completely blind.
