New 3DMark03 Patch 330

[Hellbinder]

Actually I think this is exactly what Reverend needs to do. Its exactly what the doctor ordered for his recent comments about non comparative benchmarks.

Do a complete review of the product in Question. Post the FRAPS logs, and graphs for 10 Games of various types. Played in different resolutions and Quality settings. Then say. "This is what this card offers for 400$" or 300 or 200 etc.

You could look and say AWESOME DUDE!! If i get this card i could reasonably expect an average performance in Nascar of 45 FPS with 4x FSAA+8XAF. Its my favorite game!

 

[just me]

]Actually I think this is exactly what Reverend needs to do. Its exactly what the doctor ordered for his recent comments about non comparative benchmarks.

Do a complete review of the product in Question. Post the FRAPS logs, and graphs for 10 Games of various types. Played in different resolutions and Quality settings. Then say. "This is what this card offers for 400$" or 300 or 200 etc.

You could look and say AWESOME DUDE!! If i get this card i could reasonably expect an average performance in Nascar of 45 FPS with 4x FSAA+8XAF. Its my favorite game!

That'd be fine if he did it w/10 differnt comp configs. A 3.06, 3500DDR, 9700Pro avg FPS in 10 present DX8 games tells me absolutely nothing about how my (say) K6-500, PC-100, etc. will perform w/a 9700Pro. Now or in the future.

The ORB @ FM gives me a 'sliding scale' to compare tho'. I can 'see' how my rig scales in 3DMark compared to review rigs > I can NOT see that for any game > that makes most reviews useless, to me.

Gotta remember this is about the consumer. WE have to relate > not devs, coders, reviewers.

Why is Sandra so popular? Compares to see the scaling of theoretical improvements based on other tested configs. Same w/3DMarks, CPUBench2003, Dacris, GLExcess & all the other 'good' benchmarks. 'Good' from a consumers POV.

.02,

 

[Hellbinder]

Gotta remember this is about the consumer. WE have to relate > not devs, coders, reviewers.

thats the whole point of my approach.

I just cant figure out why people instantly see *problems* with the Frpas method. When you currently see benchmarks they are only ever done on the latest UBER systems. So what difference does it make?? the FRAPS method still gives you the most realistic real world performance analasys.

 

[just me]

I can't mimic fraps results unless I own those games & even if I do > my results will vary due to a lesser CPU/DDR/card. How can I tell if I need a card &/or all the rest to achieve those FPS? I can't.

FM has a database I can go to see the sliding scale affects of CPU/DDR/FSB/etc.

Present 'game' benches w/system specs above mine are worthless & I wish reviewers would stop using their demo mobo's/CPU's/DDR to test the cards. Get the 'real world' 2.0gig, PC2700, etc. rigs out if ya want 'real world' results the majority of users can relate to. SOME reviewers do & I really like it.

Gimme a 3.0, 3500, etc. review w/any card presently out & the 3DM score & I can scale a result for any other system set-up (w/same card) using 3DM & it's massive database. I can NOT do that w/fraps.

 

[Miksu]

Just saw someone at other forum who had tested Mandelbrot with 5800U and 9700Pro. They said that they noticed same problems with R300+OpenGL as with NV30+DirectX. NV30 worked nicely with OpenGL.

 

[LeStoffer]

I wasn't going to join in this, but a series of contacts from institutional collegues have left me staggered. Staggered by the level of savvy from people with marginal 3D background, but with heavy computer industry involvement. What's the saying...? "...can't fool all of the people all of the time..." This farce will stain a particular IHV for some time, no matter how it's apportioned right now... Numerous OEMs are unhappy for various reasons. Rather unfortunate...

Very good point, stevem. It's interesting to see how everybody seems much more interested in expressing their own private feelings about this mess than discussimg how this might alter the IHV game:

The sentiment from the big OEMs means a hell of a lot more than what either Tim Sweeney or John Carmack says - or how the 3D enthusiat core feel about this.

Yes, I'm talking about nVidia's ability to maintain their clear market lead with all the implications that follows (like who defines where the mainstream hardware level is for the game producers to target, how fast we move towards cinematic rendering etc.).

I would assume that nVidia keeps their lead for now (no re-alignment within the big OEMs), but the Santa Clara based company is getting close to a three-strikes-and-you-out: The late FX range with the NV30 fiasco being the first, and this 3Dmark03 being the second.

We don't know what's going on being the scenes in this business bemongst the OEMs, and I'm quite frankly much more interested in getting to know that, that hearing what anybody on the outside says/feels.

Sorry to interrupt, but stevem's point really desires a lot more thought.

 

[chavvdarrr]

there is a review of 9800 and detailed info on 3dmark v.320 and v.330 - with animated gifs: http://ixbt.com/video2/herc-r9800-r7500.shtml
Is anyone able to shed some light onto these 2 gifs:
1) http://ixbt.com/video2/images/r9800pro-2/g3-800-330.gif
2) http://ixbt.com/video2/images/r9800pro-2/g2-840-330.gif
These are from Game3/game2 and show differences in rendering between NV35/R350 in 3dMark2003 v.330 .
Anyone who can donate referance rasterizer shots of these scenes? I think months ago (2?) maybe here there was conversation about 2nd shot, but maybe I'm wrong.

 in short ixbt&digit-life will use 3d2k3 but without analisys/details - just numbers
"no more we can be sure what we see - is it real, or cheat, or punishment from FM"

 

[Dave Baumann]

The difference is small in the PS2.0 test in some cases, purely because its such a small shader that precision doesn't make much difference. However, there are clealy differences in the wood grain effects the 5900 and 5800 shows sever black dots indicating that there is actually issues running in FP32.

 

[Marc]

The difference is small in the PS2.0 test in some cases, purely because its such a small shader that precision doesn't make much difference. However, there are clealy differences in the wood grain effects the 5900 and 5800 shows sever black dots indicating that there is actually issues running in FP32.

Do you have this black dots with 44.10 ?

 

[Joe DeFuria]

"They should stop reviewing NVIDIA cards"
See above reply: In short--a hardware review site not reviewing hardware is a bit silly.

Perhaps, but isn't a hardware review site not using an industry standard reviewing tool also a bit silly?

I thought a hardware review site was not just supposed to "review" hardware, but review it as accurately as possible. What's the point otherwise?

"They should support the 330 version".
Well, that's a matter of opinion. They apparently don't trust it yet.*

Yes, it is a matter of opinion. Again, my point is, they apparently trust nVidia drivers MORE than 3DMark, despite the fact that FutureMark is the only party to actually DOCUMENT and state its case with damning evidence. nVidia has not refuted those claims other than to say "they must be holding a grudge."

"They're not punishing nvidia"
I really don't see that they're not punishing NVIDIA. They called em cheaters, they stated they expect to have them continue cheating in the future. [/b]

How does this punish nVidia? "Yeah, they cheated...but we applaud the way they did it! You punish nVidia by not giving their hardware air-time.

You at LEAST give FutureMark the benefit of the doubt, as they at least documented the evidence.

But I presume they'll investigate closer or be skeptical of results for a while--but they can't stop reviewing the cards as that is their life blood.

What makes you think they'll be more skeptical of nVidia's results? You know, if they would USE 3DMARK, that would be one way to help them investigate, don't you think?

Reviewing cards isn't their life-blood. It's reviewing cards ACCURATLEY that is. If your readers don't TRUST your reviews, no matter how many of them you do, you're toast.

I can see how people are outraged and want more punishment, but I think that there's not a whole lot else that can be done as immediate punishment.*

I outlined it above. Either don't review the hardware, or review the hardware using drivers that don't exhibit cheating behavior.

But if you MUST review the hardware with cheating drivers, at LEAST use the benchmark that DEFEATS the cheats in your reviews. It's quite sad when being HONEST (using 3DMark that defeats detection) is considered "punishment" for nVidia. That just shows how screwed up this whole thing is.

"Futuremark should be rewarded because they're working hard as shown by 330"
Sure, and if they keep it up, I presume Guru3d will trust 3dmk03 enough to use it again. They said so in their blog. *

You are making lots of presumptions, Russ. I KNOW that they are not going to use 3DMark for the time being. And I don't see any justification for that action.

"Getting dinged more than the dingers"
As I said before, if you're a benchmarking company, its part and parcel of the job to ensure no cheating. They got caught with their pants down and it might take more than simply 330 to rebuild that relationship of trust with some people.

If catching and defeating cheats makes people LOSE trust, then those people are just idiots. If the cheats that are discovered, can be applied to almost ANY other benchmark (including ones that they will be using), and there has been NO inviestigation into similar cheating behavior....then using those other benchmarks is MORE risky, not less.

I can see people believing in Futuremark from the results of 330 and their white paper, and also I can see people having reservations that the benchmark is cheat free*

But I can NOT see people thinking that 3DMark 330 (with current driver sets) is MORE prone to being cheated than any other current high profile benchmark. That's the point.

"People should be more suspicious of NVIDIA's drivers"
I agree. *

Yay!

* Why are we talking about this? It has nothing to do with me interpreting Guru3d's applause clause. If I somehow said something that sparked these questions, I apologize completely.

We (me and you) are talking about this because we disagree about it (as my above opinions demonstrate).

Much of my response in this thread is restatement of my original opinion so it really doesn't do much but generate more words for me to continue.

What is your original opinion? (Honest question).

My general opinion is that 3DGuru's position (to continue to accept benchmarks of nVidia cards, and to NOT accept 3DMark benchmarks), is not even remotely defensible.

 

[RussSchultz]

Sorry Joe, I'm not going to get in a back and forth with you.

Bottom line:
I don't know why you're arguing with me over what 3dguru says--above and beyond the item of what I believe the applause clause means. I don't think "applauding their audacity" implies consent or approval of the actions; you do. We talked a little bit, shared our thoughts. I'm not going to move; you're not going to move. Lets leave it at that.

Right above the bottom line:
On the tangential discussion of Guru3d's decision to not use 3dmark03: I can appreciate the validity of the decision; you find it indefensible. We talked a little bit, shared our thoughts. I'm not going to move; you're not going to move. Lets leave it at that.

The world of opinion is all about differing ones. And mine, well...is different than yours. And I don't have a problem with that.

 

[Joe DeFuria]

Sorry Joe, I'm not going to get in a back and forth with you.

Your choice, though this response is at least a bit better than your previous rants and accusations.

Bottom line:
I don't know why you're arguing with me over what 3dguru says--above and beyond the item of what I believe the applause clause means. I don't think "applauding their audacity" implies consent or approval of the actions; you do.

We're arguing over this because you continue to misrepresent my point of view. I believe "applauding their audacity" AND at the SAME TIME dismissing 3DMark, implies consent / approval.

It's the sum total of their actions, not just their words, that implies consent.

We talked a little bit, shared our thoughts. I'm not going to move; you're not going to move. Lets leave it at that.

I'd rather leave it "at that" only after I'm convinced that you actually understand my point of view...which up to this very last post I don't believe you have. Hopefully, now you do, and if you want to leave it at that, great.

Right above the bottom line:
On the tangential discussion of Guru3d's decision to not use 3dmark03:

Thing is, the tangential discussion is a critical aspect of the "bottom line" as I see it.

I can appreciate the validity of the decision; you find it indefensible.

Question for you. I know you can appreciate that decision....but do you personally agree with it?

The world of opinion is all about differing ones. And mine, well...is different than yours. And I don't have a problem with that.

Nor do I...assuming what you are disagreeing with is actually my point of view.

 

[RussSchultz]

Your choice, though this response is at least a bit better than your previous rants and accusations.

gots to have that slap in there.

I believe "applauding their audacity" AND at the SAME TIME dismissing 3DMark, implies consent / approval.

I don't believe they are "dismissing 3dmark". You do. I've stated why not; you've stated why. I feel the overall tone of their blog was quite conciliatory toward 3dmark3(even though they decided not to use it for now), and at the same time didn't pull any punches. (And I know you're going to say I'm putting words in your mouth for this) I don't feel like they need to have a two paragraph rant against NVIDIA to show how big of cheaters they are in order to show their disapproval with their actions. You obviously need more than what 3dguru wrote to say they're disapproving--I don't(or its not so far away from my trip point that I'm going to say they're not disapproving).

I'd rather leave it "at that" only after I'm convinced that you actually understand my point of view...which up to this very last post I don't believe you have.

Well, you were wrong. I understood your position, I just don't share it.

Question for you. I know you can appreciate that decision....but do you personally agree with it?

I'm not in the position to actually make the decision. HOWEVER, I think that, if I were, I would also eschew 3dmk03, plus all of the currently popular benchmarking tools.

One--because everybody else uses them. If my money was earned by getting hits, I'd do my damnedest to provide different content than other websites. That would one sure fire way of differentiation. I'd try for community written benchmarks, and a wide variety of them. If I couldn't generate enough content that way to justify my time, I'd just not do it at all.

Two--for the general reasons stated(everybody targets them, and I'm not sure I trust them to be clean). This includes 3dmk03; quake3; doom3; ut2k3; At the very least not use the standard timedemos (and, since there's nothing but the standard timedemos in 2dmk03...).

EDIT: Would I continue to benchmark NVIDIA cards? Yes, or my hits go bye bye. I'd just try to be more vigilant in assuring my benchmarks aren't being subverted.

 

[Joe DeFuria]

Sorry Joe, I'm not going to get in a back and forth with you.

That didn't last long...

gots to have that slap in there.

Yeah, well, wouldn't need the occaisional slap if you wouldn't do things like F'CK'N YELL based on a pre-conceived, and wrong, perception of my view point.

I don't believe they are "dismissing 3dmark". You do.

It is a fact that they are for at least the time being not using 3DMark. That is what I mean by "dismissing" 3DMark. It will NOT be part of their reviews until further notice.

I don't feel like they need to have a two paragraph rant against NVIDIA to show how big of cheaters they are in order to show their disapproval with their actions.

Sigh. Nor do I. I thought you got that by now.

You obviously need more than what 3dguru wrote to say they're disapproving--I don't(or its not so far away from my trip point that I'm going to say they're not disapproving).

I do need more than they wrote...but I wouldn't care if they yelled from the roof-tops and dedicated 5 pages of text to how "evil" nVidia is. That's just lip-service. If you disapprove of something, you take action if you're sincere.

If in the end, they don't take any "action" against nVidia (such as simnply WAITING for different drivers from nVidia, much like they are "waiting" for some resolution for 3DMark), they didn't do something like invalidating previous reviews with these dets, or something similar. At the same time, they take action AGAINST FutureMark (stop using it).

Well, you were wrong. I understood your position, I just don't share it.

Um, you said:

I don't think "applauding their audacity" implies consent or approval of the actions; you do. "

How many times can I say that's NOT my position? So you either did (and do?) not understand my position, or you're not stating it correctly.

Question for you. I know you can appreciate that decision....but do you personally agree with it?

I'm not in the position to actually make the decision. HOWEVER, I think that, if I were, I would also eschew 3dmk03, plus all of the currently popular benchmarking tools....

And that would indeed at least be a defensible position, IMO. (Even though I would disagree with that as well.) Had 3DGuru siad something like. "well, we won't use ANY high profile benchmarking tools, including 3DMark, for the time being...." I could understand that.

But that's not what they did now, is it?

So, in your round-about-way, your actual answer to my question appears how I would answer it: that 3DGuru's decision to drop 3DMark, and only 3DMark is not something that we would agree with. Why don't you just directly answer my question, and say so?

 

[nelg]

You cannot defend the undefendable . What Guru3D (having a Nvidia bias IMO) has done is the only option left to it. By making light of what Nvidia has done it reduces the moral implications of its action. I have J- walked before. Charles Manson killed lots of people. We are both law breakers but to not draw a moral distinction is to obscure the actions in the first place. Their reaction to stop using 3Dmark03 is wrong. They now find themselves on a very slippery slope. If Nvidia is found to be cheating on other benchmarks will they also stop using them. At which point will they draw the line and stop testing Nvidia cards ? Perhaps they should make a policy of only testing Nvidia cards with workstation drivers, if these do not include cheats.

 

[Dio]

You cannot defend the undefendable .

But can you expect the unexpected?

 

[Joe DeFuria]

/ looks to nelg:

 

[Reverend]

<zips past the discussion room and yells, very quickly>
All sites who are not beta members should stop talking about using or not using 3DMark03 due to this mess. They haven't experienced the cheats, so they are not in a good position to make a truly informed decision (about whether to use 3DMark03 or not).
<zips to bed>

 

[Bjorn]

<zips past the discussion room and yells, very quickly>
All sites who are not beta members should stop talking about using or not using 3DMark03 due to this mess. They haven't experienced the cheats, so they are not in a good position to make a truly informed decision (about whether to use 3DMark03 or not).
<zips to bed>

I don't know. Since they should be more unsure about this situation, wouldn't it be more understandable if they decided to stop using it for now just for that reason ?

Edit: As for me personally, i wouldn't stop using it. The little interest i had in 3D Mark before this "cheating" thing started has been raised by an order of a magnitude. Now it's actually fun to see what happens in it between driver versions with regards to performance/image quality.

 

[Dave Baumann]

I don't know. Since they should be more unsure about this situation, wouldn't it be more understandable if they decided to stop using it for now just for that reason ?

the question is, if they stop using that, should they not also stop using everything else? We've seen what the IHV's are capable of doing here, and it applies just as much to a game benchmark as 3DMark except that we have people policing 3DMark - who is policing gamebenchmarks?