Motherboards.org uses 53.03 benching 3dm2k3?!?

[digitalwanderer]

I appreciate and respect everyone’s opinions on the matter and acknowledge that you don’t agree with some stuff. I know that some people will disagree, and some people will agree, and some people just won’t say anything.

I’m not asking you to agree with us; all I am asking is that you understand our focus and how we are evaluating gameplay performance and IQ to make up the gaming experience.

We use games, and use FRAPS, and actually play the game. If we can create a recorded demo in the game we do so for accurate results, but we do not share the recorded demo. In fact, take for example Tomb Raider AOD, I created a recorded demo in it that only I have, it only exists on my hard drive, not even Kyle has it. So we keep these recorded demo's private. In the case that a game can't create one we do a manual run through in a certain place where we can guarantee the same results over and over again. These things about gameplay performance and IQ we take very seriously in our reviews.

No argument from me on the way you do your reviews or what you focus on, I've really liked the last few you've done with games and fraps. I think fraps & gaming IS the best way to review, but I also feel that synthetics tell a lot too and are important.

I just disagree with [H]'s stance on 3dm2k3 in a big way, but I ain't gonna bash and get John all ornery at me.

 

[martrox]

While I totally disagree with the original reasons for [H] approach, and with the way Kyle handled himself, at this point I do agree with exactly what they [H] are doing. The real villain here is nVidia and their cheating ways, but the fact is that Futuremark is now getting what they deserve. Rather that take a stand and hold to it, they played politics and it has come back to bite them in the ass. IF they had maintained their original assertions that nVidia was cheating, had taken a hard line and not given in, they at least would have been looked at as a very principled company, whose benchmark at least told the truth about "future"capabilities where shaders are concerned. At this point, all you can say about Futuremark is that, if anything, they have helped nVidia through their own incompetence.

 

[Brent]

I appreciate and respect everyone’s opinions on the matter and acknowledge that you don’t agree with some stuff. I know that some people will disagree, and some people will agree, and some people just won’t say anything.

I’m not asking you to agree with us; all I am asking is that you understand our focus and how we are evaluating gameplay performance and IQ to make up the gaming experience.

We use games, and use FRAPS, and actually play the game. If we can create a recorded demo in the game we do so for accurate results, but we do not share the recorded demo. In fact, take for example Tomb Raider AOD, I created a recorded demo in it that only I have, it only exists on my hard drive, not even Kyle has it. So we keep these recorded demo's private. In the case that a game can't create one we do a manual run through in a certain place where we can guarantee the same results over and over again. These things about gameplay performance and IQ we take very seriously in our reviews.

No argument from me on the way you do your reviews or what you focus on, I've really liked the last few you've done with games and fraps. I think fraps & gaming IS the best way to review, but I also feel that synthetics tell a lot too and are important.

I just disagree with [H]'s stance on 3dm2k3 in a big way, but I ain't gonna bash and get John all ornery at me.

hehe, i'll stop too

if anyone ever has any input or opinions don’t hesitate to email me, i'll listen to all feedback and criticism, i really enjoy learning and growing, i sure as hell don't know everything and will be the first to admit it, everyday i learn something new in this great world wide web we live in

 

[jb]

Brent,

my biggest issue and not directed at [H] (as 3DGPU can be lumped into this category) is what sites like [H] are you doing to the guy that can only upgrade once every 2 years. Yes current games give a decent picture of what to expect today. But does current games tell you what to expect in 6 months, 9 month, 18 months? How is this guy gonna to have any confidence his card he buys today wont have some limitations do the tech the next generation will be using? Current games can not expose weakness in future technology that the new games will use. Synthetics can or at least give the use some idea of where to cards strengths/weakness are. And we all know that a score of xxxx 3dmarks = Nothing of real value. But at least you can see that one set of cards has a strength/weakness in a particular area. And maybe that's worthy info for the once every 2 years upgraders. Again I don't dislike what you folks are doing...but its more of less of putting blinders to what is coming in the future and IMHO that's not the best way to handle it. We have all seen reviews of the 56xx cards that have said they will be fine for future games. However we all know that is not going to be the case if the user runs DX9 in any detail.....

 

[ClyssaN]

Brent,

my biggest issue and not directed at [H] (as 3DGPU can be lumped into this category) is what sites like [H] are you doing to the guy that can only upgrade once every 2 years. Yes current games give a decent picture of what to expect today. But does current games tell you what to expect in 6 months, 9 month, 18 months? How is this guy gonna to have any confidence his card he buys today wont have some limitations do the tech the next generation will be using? Current games can not expose weakness in future technology that the new games will use. Synthetics can or at least give the use some idea of where to cards strengths/weakness are. And we all know that a score of xxxx 3dmarks = Nothing of real value. But at least you can see that one set of cards has a strength/weakness in a particular area. And maybe that's worthy info for the once every 2 years upgraders. Again I don't dislike what you folks are doing...but its more of less of putting blinders to what is coming in the future and IMHO that's not the best way to handle it. We have all seen reviews of the 56xx cards that have said they will be fine for future games. However we all know that is not going to be the case if the user runs DX9 in any detail.....

I share the same thought

 

[digitalwanderer]

I ain't gonna bash and get John all ornery at me.

hehe, i'll stop too

 

[Althornin]

If you understand the focus of our site when it comes to evaluating video cards now you will see that decision makes sense. We cater to the gamer, and to the gamer the only thing that relates game performance are games themselves. Our goal is to evaluate gameplay experience, and that can only be done with games. That is what HardOCP has chosen to do.

Brent, i understand the focus of your site.
Waht i dont understand is how you can shrug off the predictive benefits of synthetic benchmarks - they can predict how future games will run, and most gamers want to know how thier card will handle future games as well!
Please, dont ignore this, or consider it a bash, i just dont get it. I'd like some evidence that sythetic benchmarks fail to predict future game performance (which is gonna be hard to come by, because all the shadermark/3dmark tests let us predict performance in games like TR:AOD just fine...) - if thats not your reason, then what is?

Apparently you think gamers cannot understand how to predict future performance? Or what?

 

[Ali]

Does it matter if [H] doesnt do the synthetics? If you are looking for a graphics card review, then if you only go to one site, you deserve what you get.

If you want technical details, come here, if you want game data, goto [H], if you want nVidia bias, go to Toms. Seems easy and simple to me.

The thing that worries me though is nVidia working out some way to cheat fraps. Their drivers already have pop-up protection, whats to stop them adding a fraps detector, and changing its output?

If we as a community dont deal with this cheating by nVidia right now, when ther is clear evidence against them, then we are just setting ourselves up to get shafted further down the line.

By [H] attacking futuremark (well, kyle), and not nVidia, then the general gaming public ([H] target market) will never hold cheating against nVidia in the future, and nVidia wont have any reason to think twice about cheating, as they get away pretty much scott free.

Just my 2cents.

Ali

 

[AzBat]

Yeah, I'm just getting to the party.

I figured I would voice my opinion on these matters. A few of you might find value in them. If not, then tough, I'm giving them anyway!

I've sent Brent an email on my thoughts about [H] and 3DMark. I won't repeat them here, but I will say I don't think [H]/Kyle has treated Futuremark/3DMark fairly.

As for the rest of this, I can understand that some sites are pissed that 3DMark are asking them to remove benchmark results using un-approved drivers. However, if they were doing their job right in the first place then Futuremark wouldn't need to ask them to remove them. I think everybody should continue notifying Futuremark when they find sites using unapproved drivers.

I believe 3DMark is in the right by asking these sites to remove the results using unapproved drivers. It's their software and the users do agree to all the agreements for using such software. These sites are just going to have to do more work and make sure that they are following the guidelines in the future. If they want to complain they need to go to NVIDIA as they are the ones not releasing approved drivers.

Whether or not Futuremark could be doing a better job is a totally different issue together. Is there room for improvements? Yes. Is the current method not working? I don't think so. I do feel that's it's going to be even more important that 3DMark makes sure reviewers adhere to the usage guidelines. Especially with next generation hardware just around the corner. Can they continue using the same method? Probably, but I believe it would be easier with some improvements. Like others have said it's hard for reviewers to use set of drivers for the review and then have to use a totally different set for 3DMark. The bad thing is it's possible that reviewers would rather quit testing with 3DMark then have to use 2 different drivers. Although, I believe that's lazy and not something I would attribute to a respectable site. What improvements need to be made? I've seen some good ones. Unfortunately I'm not sure requiring an Internet connection to Futuremark is going to work. Like others have said most reviewers probably don't have the machines connected to the Internet while running the tests, I know I never did. But eventually I would connect once the tests were done. I guess you could make 3DMark not display the results until the machine was connected to their servers. This would in effect get rid of the local Results Browser and rely solely on the ORB. However it is done, I believe incorporation of approved drivers into the software is going to be required. It's pretty sad if they have to do this, but what can you do when people are too lazy to go to the web site to see what's approved? At the very least they still need to list which drivers have been tested on the approved list. It doesn't make sense to leave it ambiguous as to whether the drivers have been tested/approved. Gives people an excuse when they happen to use unapproved drivers(ex: overclockcafe.com).

Joe and others have suggested that Futuremark should release a new patch every time a new driver is released. To me that's not a good answer. That puts more work on Futuremark for something they didn't create. Plus, there is already going to be a lapse between the release of a new driver and a patch to kill the cheats. They can't and shouldn't get the drivers in advance. Plus, they have do a lot of testing to make sure they caught all the cheats. We've already seen this isn't always possible. Then they have to go through a QA process to make sure the patch is good. In the end, I believe the current action is correct. They are penalizing IHVs by not approving every driver release. If people start seeing that a IHV is not being fair and people quit buying or using their products then eventually the company will be forced to play fair.

Anyway, I'm hoping 3DMark releases 3DMark04 soon. I'm thinking the longer they keep using 3DMark03 the more it contaminates their image and makes people question the validity of future updates.

Tommy McClain

 

[Joe DeFuria]

Joe and others have suggested that Futuremark should release a new patch every time a new driver is released. To me that's not a good answer.

PPttthhhhhtt!

That puts more work on Futuremark for something they didn't create.

While that is certainly true, that's what companies do to protect their product, and their customers confidence in their product. Microsoft doesn't "create" security problems...they get taken advantage of and exploited by purposeful hackers. Yet, I expect Microsoft to actively protect their OS rather than tell everyone / would be hackers...."just don't do that!"

Plus, there is already going to be a lapse between the release of a new driver and a patch to kill the cheats.

There is also a lapse between when a web site posts invalid benchmarks, and when and if they are removed. There's no real way around the time lapse issue, no matter which path is taken IMO.

They can't and shouldn't get the drivers in advance.

Actually, I think they should. I think FM should require participants to send them WHQL candidate drivers at the same time they are sent to MS for certification.

Plus, they have do a lot of testing to make sure they caught all the cheats. We've already seen this isn't always possible.

Yes, it is a lot of work, but at least there is a definite pay-off: once you release the patch, the results are valid. I'm not sure it isn't possible...I have no idea what FM's testing methodologies are, how time consuming they are, etc. Again, it definitely is a lot of work though.

They are penalizing IHVs by not approving every driver release.

Problem is (as you've said), it seems like they are penalizing web reviewers more than IHVs. I think this is putting people off from using 3DMark.

If people start seeing that a IHV is not being fair and people quit buying or using their products then eventually the company will be forced to play fair.

I just don't see it happening. I see it continuing just as it is now: nVidia releases the latest and greatest drivers...web sites put up reviews with 3DMark scores with them....wash, rinse, repeat.

Anyway, I'm hoping 3DMark releases 3DMark04 soon. I'm thinking the longer they keep using 3DMark03 the more it contaminates their image and makes people question the validity of future updates.

Unfortunately, unless nvidia's hardware is actually better than ATI's hardware (without cheating) on '04, then I suspect the issues will continue.

 

[jb]

While that is certainly true, that's what companies do to protect their product, and their customers confidence in their product. Microsoft doesn't "create" security problems...they get taken advantage of and exploited by purposeful hackers. Yet, I expect Microsoft to actively protect their OS rather than tell everyone / would be hackers...."just don't do that!"

Well big difference. MS has the means to do this. FM doesnt. Means == Money!!!!!

 

[Joe DeFuria]

Well big difference. MS has the means to do this. FM doesnt. Means == Money!!!!!

And therin lies the rub.

If FM doesn't have the money to invest to do it, I fear that 3DMark won't be around too much longer. People just won't use 3DMark if they don't trust it, or if it's too much of a hassle (load different drivers.)

 

[digitalwanderer]

Plus, there is already going to be a lapse between the release of a new driver and a patch to kill the cheats.

There is also a lapse between when a web site posts invalid benchmarks, and when and if they are removed. There's no real way around the time lapse issue, no matter which path is taken IMO.

Couldn't FM pre-make a generic patch? I'm not sure exactly what they do when they patch it, but couldn't they just make the executable slightly different and rename the shaders or something in a generic fashion which would bypass app detection/shader detection? Then they could just release a new patch whenever nVidia released new drivers, or at least test the new drivers against the new patch.

Or the "submit drivers to FM when you submit 'em for WHQL" is a good idea too.

ANY solution is going to involve work on FM's part, but that's part of the price of making their product reliable.

 

[Doomtrooper]

Since Futuremark has already released its criteria for approved drivers if they truly cared about authenticity they would simply not allow uploads for non approved drivers. This fixes alot of things, and since these OEM deals are usually obtained by having the superior '3Dmark' score (popularity of a certain brand due to high 3Dmarks in other words), this goes right to the source of the cheating.

Lets face it DELL will want to equip their machines with a popular HIGH 3Dmark graphic card for them to make sales too, although it surprises me that they have not got complaints about the piss poor performance of the Nv30 series.

 

[AzBat]

Joe and others have suggested that Futuremark should release a new patch every time a new driver is released. To me that's not a good answer.

PPttthhhhhtt!

Back at you buddy! All these comments and you didn't even comment on the rest of my message. LOL

That puts more work on Futuremark for something they didn't create.

While that is certainly true, that's what companies do to protect their product, and their customers confidence in their product. Microsoft doesn't "create" security problems...they get taken advantage of and exploited by purposeful hackers. Yet, I expect Microsoft to actively protect their OS rather than tell everyone / would be hackers...."just don't do that!"

Like jb said, Futuremark doesn't have the means like Microsoft. We've seen what they were able to invest to fix the problem and it's not something that I would say cost a lot of money. Bowing down to lawyer pressure was enough to tell you they don't have adequate resources to really fix the problem.

Plus, there is already going to be a lapse between the release of a new driver and a patch to kill the cheats.

There is also a lapse between when a web site posts invalid benchmarks, and when and if they are removed. There's no real way around the time lapse issue, no matter which path is taken IMO.

Agreed. So why even bother with a new patch? It don't make sense if you ask me. Provide one update and if the IHVs don't abide to the guidelines, then fuck'em.

They can't and shouldn't get the drivers in advance.

Actually, I think they should. I think FM should require participants to send them WHQL candidate drivers at the same time they are sent to MS for certification.

I can see where this is appealing, but going from past experience it's a VERY BIG mistake. What happens when they announce that they have tested the drivers the day they are released and everybody sees they're not approved? Everybody is going to be crying fowl including the offending company. People are going to say well you just released a new patch that disables valid features like the unified compiler. Or if the patch and driver combination increase the results, then they will say you did something in the patch that makes them look better. They need to write one set code and let all drivers current and future run on it. Too many patches just dilutes the relevancy of the benchmark.

Plus, they have do a lot of testing to make sure they caught all the cheats. We've already seen this isn't always possible.

Yes, it is a lot of work, but at least there is a definite pay-off: once you release the patch, the results are valid. I'm not sure it isn't possible...I have no idea what FM's testing methodologies are, how time consuming they are, etc. Again, it definitely is a lot of work though.

The only one that wins from patches are the cheating IHVs not anybody else. Futuremark stands to lose if they invest all that time and then find out that cheats still got by. Why devote all those resources to a year-old benchmark that had a year where its imaged is tarred and the replacement is just around the corner? The economics just don't make sense at this point in it's life. Made sense the first 6 or so months, but not now.

They are penalizing IHVs by not approving every driver release.

Problem is (as you've said), it seems like they are penalizing web reviewers more than IHVs. I think this is putting people off from using 3DMark.

That's been the outcome so far, but I believe that could have been lessened if Futuremark had done a better job getting the word out to reviewers at what they expect and if they would have been clearer which drivers were tested and/or approved. Had they done that I'm not sure reviewers would have been penalized as much.

If people start seeing that a IHV is not being fair and people quit buying or using their products then eventually the company will be forced to play fair.

I just don't see it happening. I see it continuing just as it is now: nVidia releases the latest and greatest drivers...web sites put up reviews with 3DMark scores with them....wash, rinse, repeat.

Sad state of the reviewing industry if that's the case. I think it's just as much the reviewers fault as it is the cheating IHVs fault. It's the reviewers responsibility to keep the public informed about wrongful deeds by the IHVs. If they're not doing their job right, then the consumer will always be easily manipulated by the IHVs.

Anyway, I'm hoping 3DMark releases 3DMark04 soon. I'm thinking the longer they keep using 3DMark03 the more it contaminates their image and makes people question the validity of future updates.

Unfortunately, unless nvidia's hardware is actually better than ATI's hardware (without cheating) on '04, then I suspect the issues will continue.

Agreed, but I'm thinking or at least hoping that Futuremark has got some better ideas of making it a non-issue by writing the software in such a way to make it more difficult for NVIDIA or whoever else to try and cheat. I guess we will see, huh?

Tommy McClain

 

[Brent]

Brent, i understand the focus of your site.
Waht i dont understand is how you can shrug off the predictive benefits of synthetic benchmarks - they can predict how future games will run, and most gamers want to know how thier card will handle future games as well!

Please, dont ignore this, or consider it a bash, i just dont get it. I'd like some evidence that sythetic benchmarks fail to predict future game performance (which is gonna be hard to come by, because all the shadermark/3dmark tests let us predict performance in games like TR:AOD just fine...) - if thats not your reason, then what is?

Apparently you think gamers cannot understand how to predict future performance? Or what?

Well, you see, while synthetics may be able to tell you which card is faster at whatever PS version etc.. That doesn't necessarily mean which card will be faster in an actual game. Developers and drivers from IHV's do use valid optimizations to increase performance. Sure, the synthetic may tell you the raw power, but it doesn't always work out to mean the same thing when it comes time to test a game.

Here is a hypothetical example, Tomb Raider AOD uses PS 2.0. The 9600XT in PS 2.0 according to synthetics vs. the 5700U shows a big difference. However, in the game the difference of the minimum FPS of a recorded demo in a level is only on the order of 4 FPS in difference. The percentage of the difference in the FPS in the game is less than the percentage of the difference in the synthetic tests. So according to the synthetic tests the 9600XT should be a lot faster in TRAOD, but in actual reality the gameplay performance isn't that much faster.

See what I'm getting at, the synthetics will tell you the raw performance differences, but in a game it doesn't always parallel with the synthetic tests, for whatever reasons.

I'm not shrugging off synthetics, I'm just saying synthetics don't represent gameplay experience, and gameplay experience is the focus of HardOCP.

The Direct Question: "How do current games predict future performance and longevity of a video card?"

The short answer, they don't. They can give you an idea because a future game may use the same gaming engine, but as everyone knows just cause it uses the same engine doesn't mean it will perform the same. So what can you do? Pretty much stay on top of the game and make sure you are using the latest most popular games out there using the most features, i.e. what will become heavy feature usage like heavy shader usage and the like and which gaming engines will be used the most.

Now, me personally, I like synthetics, even Kyle will tell ya, I always use to include synthetics in the reviews, I've used the PS Precision test, I've used Shadermark from the very beginning, and of course I've been using Futuremark/Remedy since Final Reality came out a long time ago! I even started to use Humus's demo's and Demo Scene demo's. I agree that they have a use and I do believe they have a place.

However, that place is not at HardOCP. The focus of HardOCP's reviewing of video cards has changed, and the focus is now on gameplay experience, and so I have taken up the reigns and heading that up at HardOCP and I think we've done a great job so far and I will continue to focus on games and the experience they deliver for as long as I work at HardOCP.

There is no wrong or right here IMHO, just simply a different method. One you may not agree with, and that’s perfectly fine.

I’m glad we could have a nice mature discussion about this, I hope I’ve cleared up some things and hopefully everyone understands, while you may not agree with it, at least maybe you can understand it some. Anyways, I’m done now, really Gotta get back to work, got a review to finish If you have any further comments feel free to email me with them.

Love you all! L8r

 

[AzBat]

Since Futuremark has already released its criteria for approved drivers if they truly cared about authenticity they would simply not allow uploads for non approved drivers. This fixes alot of things, and since these OEM deals are usually obtained by having the superior '3Dmark' score (popularity of a certain brand due to high 3Dmarks in other words), this goes right to the source of the cheating.

Lets face it DELL will want to equip their machines with a popular HIGH 3Dmark graphic card for them to make sales too, although it surprises me that they have not got complaints about the piss poor performance of the Nv30 series.

DT,

I know you've been vocal about this and at first I thought Futuremark was correct in how they initially handled it(segregating the results). However, I'm not so sure now. I can see where Futuremark wants all results to be submitted to them whether or not they used valid drivers. The data still can be useful to others that post using the same drivers and internally Futuremark can dissect the data to find other uses for it. But like you said they should care about the authenticity of their results, especially the data that is being used by the public from the ORB. However, by accepting results from non-approved drivers they're diluting the relevancy of the accurate results and making it that much easier for people unintentionally getting mislead by results that may not be accurate. Unfortunately, I don't see your idea fixing the problem with reviewers running the benchmark and then posting their results on their web site with unapproved drivers. The only thing that's going to fix that is if the software itself doesn't give them the results when using unapproved drivers.

Tommy McClain

 

[Doomtrooper]

My idea is to fix the benchmark so it does not mislead people, and also not allow inferior chipsets flooded to the market by having a accurate score.
Reviewers will always do what they please, there is no 'controlling' them...I would hope most would use common sense, but most reviewers seem to be just looking for free hardware.

 

[ByteMe]

JRR: You can't say you weren't warned.

 

[ben6]

The one thing I hate is any review that says "buy this card for DX whatever games". I can count the number of true DX9 games released on one hand, and frankly I'm disappointed by each and every one of those games except maybe Tron. Not saying [H]ardOCP or anyone here is guilty of that, but too many sites are.

Tomb Raider Angel Of Darkness
Halo
Deus Ex Invisible War
Gunmetal
Tron 2.0

Did I miss any?