How benchmark 3D Cards, today

[FrgMstr]

I don't doubt it. Resource realites are resource realities. Doesn't change the answer to the question tho.

Unless you live in the land of the fairies..... The fact of the matter is that time and money resources DO give you the answer in the real world. Theoretically I would have done a lot of things differently over the last 8 years, but the fact is that we simply have not had the resources to accomplish all of those lofty goals. So we work inside of our means. Do we make concessions? Yes we do.

So you can sit here and theorize just exactly how it SHOULD be done, but that just may not be a solution that is realistic.

 

[Geo]

Well, this is what I was responding to: http://www.beyond3d.com/forum/showpost.php?p=712811&postcount=93

I already said a periodic visitation of the issue is a reasonable compromise given the resource limitations. I can't look around and find anyone else who's doing what I'd like, so I certainly can't fault [H] for failing to meet a standard.

But I don't have any problem between knowing what I'd like to see on the one hand and what it is reasonable to expect at any given moment on the other. Indeed, it's kinda hard to move over time to transition the first into the second without that knowledge. . . It's what roadmaps are all about, right?

 

[FrgMstr]

Well, this is what I was responding to: http://www.beyond3d.com/forum/showpost.php?p=712811&postcount=93

I already said a periodic visitation of the issue is a reasonable compromise given the resource limitations. I can't look around and find anyone else who's doing what I'd like, so I certainly can't fault [H] for failing to meet a standard.

But I don't have any problem between knowing what I'd like to see on the one hand and what it is reasonable to expect at any given moment on the other. Indeed, it's kinda hard to move over time to transition the first into the second without that knowledge. . . It's what roadmaps are all about, right?

Roger that, fully understood. I think we are on the same page here.

 

[GrapeApe]

..Only used NVIDIA. BFGTech sponsored the article. We stuck with one brand as it is my opnion that it will give us results..

And also keeps the sponsors very happy I'm sure. And considering...

This has been a tremendous amount of work and required a lot of time and money to get done.

Ever think of getting ASUS or Gigabyte (both sellers of both ATi and nV) to sponsor? Or maybe to them sponsorship isn't as vital because they know that molding the reviews and the community isn't as important because in the end they have you regardless of which camp people choose.

While I agree that it's easier to avoid the optimization/IQ debate the way B3D does with their reviews of just ATi and nV cards, but if you're trying to scale the cards and cpu you don't think people would accept a little compromise, especially since there hasn't been much focus on the issue in prior reviews for them to care in the first place?

I guess page-hits and advertising alone brought from good reviews isn't enough to avoid the need for 'sponsored reviews'. However I'd never take one of those any more seiously than I would the Pepsi challenge. Why should I think you (or anyone else, doesn't matter whom) wouldn't omit a poor result for the whole line if you should come across a performance hole?

Too bad there's no equivalent Consumer Reports for tech, unfortunately everyone in the hardware industry is worried about all the PR money out there they can get a chunck of and they risk losing.

 

[Mendel]

You should tr tp pitch it as a hit collector...

Eh, what?

 

[Geo]

We have a CPU / Video Card scaling article that is nearing completeion now. 6 different CPUs, 3 from Intel, 3 from AMD. Dual core included in both. 6 or 7 video cards used. Only used NVIDIA. BFGTech sponsored the article. We stuck with one brand as it is my opnion that it will give us results that are easier to represent without a bunch of argument on the video card end. We have used real-world gameplay and IQ scaling as the measure for comparison as we do in current video card evaluations.

This has been a tremendous amount of work and required a lot of time and money to get done.

And thar' she blows!: http://enthusiast.hardocp.com/article.html?art=MTAwMiwxLCxoZW50aHVzaWFzdA==

Edit: I understand how sponsorship would be helpful (duh), and have no reason to question the statement at the top that BFG had no input or "preview" of the results. But I also can't help but note that there have been suggestions in the past that ATI and NV might have some differences in the degree to which the drivers rely on the cpu in some instances. . .and it would have been nice to have had the opportunity to test that theory. Ah well.

EditII: Tho having quibbled, still appreciate having the article at all.

 

[FrgMstr]

And thar' she blows!: http://enthusiast.hardocp.com/article.html?art=MTAwMiwxLCxoZW50aHVzaWFzdA==

Edit: I understand how sponsorship would be helpful (duh), and have no reason to question the statement at the top that BFG had no input or "preview" of the results. But I also can't help but note that there have been suggestions in the past that ATI and NV might have some differences in the degree to which the drivers rely on the cpu in some instances. . .and it would have been nice to have had the opportunity to test that theory. Ah well.

EditII: Tho having quibbled, still appreciate having the article at all.

Yes, it would have been nice to have done that, but the fact of the matter is that it took Brent a month to cover what we did.

Another reason we went with BFG is because they are simply easy to do business with. I think the entire negotiation on the sponsorship negotiation took less than 10 minutes. [H]: "We are doing a CPU scaling article, due to time, we are going to limit video cards used. We need these cards and $X to fund the project and it will make you the only video card builder noted." BFG: "OK"

I think the article is some of the best work we have done in the last year. That said, almost NO OTHER hardware site on the net linked it (Yes, a couple did but no real mention was made of it much anywhere.) So was it just that bad, or did our real world testing and results fly in the face of their benchmarks that they have based their past opinions on?

 

[digitalwanderer]

So was it just that bad, or did our real world testing and results fly in the face of their benchmarks that they have based their past opinions on?

I think a lot of it probably had to do with the "only one video card" thing. Even if there was no impropriety and the intention was to limit the results to just the CPU it probably gave the impression of bias given some past leanings/tendancies.

Just my .02 on possibilities, I don't mean it as a dig or a troll. I didn't post it 'cause I haven't been posting news much lately, the family has got me too busy.

 

[Geo]

I think the article is some of the best work we have done in the last year. That said, almost NO OTHER hardware site on the net linked it (Yes, a couple did but no real mention was made of it much anywhere.)

Yes, that is a pity. I hope that's an indication of misreading the interest level of their readers in such comparisons. I mean, obviously I hope that as I started the digression upstream.

 

[Tim Murray]

I have to question the selection of games, though. 3 FPS games and WoW? Why not games from different genres?

 

[FrgMstr]

I have to question the selection of games, though. 3 FPS games and WoW? Why not games from different genres?

WoW because is was the #1 selling PC game in 2005 by a margin of 100%. And the rest because they are still hugely selling games that can heavily stress a video card.

http://biz.gamedaily.com/charts/?id=182

 

[FrgMstr]

Yes, that is a pity. I hope that's an indication of misreading the interest level of their readers in such comparisons. I mean, obviously I hope that as I started the digression upstream.

The reader feedback was AMAZINGLY POSITIVE. You can read our forum replies here.

http://www.hardforum.com/showthread.php?t=1031925

 

[Tim Murray]

WoW because is was the #1 selling PC game in 2005 by a margin of 100%. And the rest because they are still hugely selling games that can heavily stress a video card.

http://biz.gamedaily.com/charts/?id=182

Well aware of the sales figures for each--I just don't think that three games that stress the video card specifically is necessarily the best idea for a CPU scaling article. Why not Age of Empires 3 or Star Wars: Empire at War? Admittedly I've never played either (although I do remember AoE 3 being considered not too easy on the video card), but I think that one of those would give a more interesting spin on CPU scaling.

 

[Geo]

The reader feedback was AMAZINGLY POSITIVE. You can read our forum replies here.

http://www.hardforum.com/showthread.php?t=1031925

Well, not amazing to me. Gratifyingly, that's a good word. Hope you guys are inspired by your reader's reactions to revisit periodically. . .once a year'd be nice.

Edit: http://www.hardforum.com/showpost.php?p=1029182261&postcount=12 That's a pretty decent thumbnail of what I took out of it at the top, tho a Quake4 would have shown there is some reason to hope on the dual-core front.

EditII: As an aside, just going thru MS GDC multi-core presentation .ppt. The happy thot I took from that, and the fact that one of the presenters was an XB360 guy, is that the next gen consoles should do a lot to light a fire under this in the medium-to-longer-term.

 

[FrgMstr]

Well, not amazing to me. Gratifyingly, that's a good word. Hope you guys are inspired by your reader's reactions to revisit periodically. . .once a year'd be nice.

Edit: http://www.hardforum.com/showpost.php?p=1029182261&postcount=12 That's a pretty decent thumbnail of what I took out of it at the top, tho a Quake4 would have shown there is some reason to hope on the dual-core front.

EditII: As an aside, just going thru MS GDC multi-core presentation .ppt. The happy thot I took from that, and the fact that one of the presenters was an XB360 guy, is that the next gen consoles should do a lot to light a fire under this in the medium-to-longer-term.

Hehe, when Intel starts doing public contests that are looking for "Best Threaded Game," it is time to start worrying.
http://www.hardocp.com/news.html?news=MTc3MTYsTWFyY2ggICAgLDIwMDYsaG5ld3M

 

[poopypoo]

Well aware of the sales figures for each--I just don't think that three games that stress the video card specifically is necessarily the best idea for a CPU scaling article. Why not Age of Empires 3 or Star Wars: Empire at War? Admittedly I've never played either (although I do remember AoE 3 being considered not too easy on the video card), but I think that one of those would give a more interesting spin on CPU scaling.

I'm as confused by this as i am by your conclusion -- that dual cores make no positive difference to gamers. This is a pretty odd conclusion to draw without having a test, "BF2 with 6 filesharing apps, GAIM, and a virus scan running". Admittedly, we currently all try to avoid such situations, but isnt the whole point of dual core (for gamers) to allow us to play (at least rather dated games like WOW) without tanking if the virus scan kicks in? 9_9

 

[FrgMstr]

I'm as confused by this as i am by your conclusion -- that dual cores make no positive difference to gamers. This is a pretty odd conclusion to draw without having a test, "BF2 with 6 filesharing apps, GAIM, and a virus scan running". Admittedly, we currently all try to avoid such situations, but isnt the whole point of dual core (for gamers) to allow us to play (at least rather dated games like WOW) without tanking if the virus scan kicks in? 9_9

I would suggest what you are describing is a difference in multitasking, not gaming. We covered this in 2005 and it is very easy to show.

What if we take the same setup and place a workload on the CPU such as a DivX encode and then run our gaming benchmark? Remember that current Dr. DivX 1.06 is lightly multithreaded, so it will take up some workload on both cores.

While there is obviously a frame rate hit in our game, anyone that has ever done video encoding on a desktop computer will probably find this amazing that we could run the benchmark at all. While our game is not as fast, it is certainly “playable.â€￾

The page is located here with more data.

I also added this to the conclusion page.

Multithread & Multitask

It is important that you realize and keep your thoughts clear on this subject. “Multithreadâ€￾ and “multitaskâ€￾ are two very different things, although I sometimes hear them thrown around interchangeably. Let’s get some basics set here.


Multithreaded applications are ones that can utilize more than one processor at a time. And multitasking is using more than one application at a time. Windows XP Pro is a multithreaded aware operating system, so it can utilize an Intel 840 processor’s dual cores. This does not mean that every program running on that OS are capable of using the dual cores as well. Specific applications have to be multithreaded in order to directly benefit from dual core processors. At this time, there are very few applications that are multithreaded with the exception of desktop image and video programs.


Multitasking on the other hand is the ability of your computer to do several things at once. It can take many programs and run them side-by-side. And when we have a multithread capable operating system, it can run those single threaded apps at the same time, many times making the application usage much smoother or just simply faster as they are not fighting for CPU cycles.

Hope this helps.

 

[poopypoo]

Brent's review is ambitious and thorough, and your response to my post was also quite thoughtful. My point, however, was simply that dual core seems like a great idea right now, because even though the performance gains in "pure gaming" are well-known to be negligible, does anyone actually run gaming machines without at least AV, IM, and BT running? Throw in F@H, firewall, fileserving, and (my favorite) scheduled PVR, and you've got quite a case against single-core gaming, IMO. Sorry if my earlier comment was curt -- I have absolutely no interest in dissing Brent's great scaling article (I will probably bump my A64 o/c up a little higher in response to the results) -- I just thought that part of the conclusion seemed a little unwise, given the real-world needs of most gamers. Lol... in this, a thread on GPU review techniques (and not specifically yours), I posit that different methods of testing CPUs could be employed in your review(IMO, with benefits). ^^;

 

[FrgMstr]

Oddly enough, even though there will be little else that multi-core processors in the forseeable future, I am talking to more and more people every day that are using their desktops ONLY for gaming. I am finding that many folks are using their laptops for all their other needs, but going back to the fullsize desktop when they want to get their frag on. This is a statement to just how powerful our laptops have gotten.

 

[digitalwanderer]

Oddly enough, even though there will be little else that multi-core processors in the forseeable future, I am talking to more and more people every day that are using their desktops ONLY for gaming. I am finding that many folks are using their laptops for all their other needs, but going back to the fullsize desktop when they want to get their frag on. This is a statement to just how powerful our laptops have gotten.

Or how indispensable desktops are to gaming, depending on how you read it.