Reverend at The Pulpit #3

[Reverend]

What a week, eh?

<u>The NVIDIA and 3DMark03 issue</u>
I'd like to think this was a nightmare and that it never happened. But it did. And we, as reviewers, will need to re-think the way video card reviews are conducted as well as presented. While the current mindset is to think "NVIDIA cheating!" and "3DMark03 = useless!", I prefer to overlook the two names here and think of things in general terms. To simplify -- if this is happening with NVIDIA and 3DMark03, it can happen with any IHV with any fixed-cam mode benchmark. This includes pre-recorded gameplay demos used for benchmarking. I'm sure you've read at some website the declaration that "Benchmarks = Dead". Well, it's not because the reviewers will have to do things a bit differently now. There are various ways of doing this obviously and Dave will probably be holding a "meeting" with the rest of the crew on the way we should be doing reviews.

Dave said he was very depressed about the whole thing, what it means to Beyond3D (basically there isn't anything we can do to prevent this from happening at the source) and what it means to the consumers. I wouldn't say I'm too depressed but it is worrying because it affects the very nature of Beyond3Ds existence.

<u>DOOM3 benchmarking</u>
I don't really want to talk about this but I probably should, just to summarize all that I said.

When I read that www.hardocp.com and www.anandtech.com (btw, how often do you type mistype "anand" if you typed it quickly?!) were priveleged (and I do think it is a privelege) enough to be considered by NVIDIA for the arrangement/deal they made with id Software, I immediately shot an email to id Softwares John Carmack and Marty Stratton with this :

Hi John and Marty,

Is there any way we could work together on a similar basis as per what was accorded to HardOCP and AnandTech?

Thanks.

Marty (as I expected to reply, rather than John) responded with :

Anthony,
NVIDIA chose those outlets - we did not. It would be best to place this
request with them.

Thanks!
Marty

... to which I immediately replied with :

Thanks for responding Marty. I'll see if NVIDIA comes through for Beyond3D.

Now, where are those new never-before-seen screenshots you said you *may* give me?

Seriously though, thanks.

My initial thoughts, which weren't expressed in my email directly above, was :

1) Marty said "outlets", which I took to mean "websites"
2) id Software do not want to be bothered with websites making direct approaches with them, for the sole reason, in my opinion, that id Software aren't too informed of which hardware website (as opposed gaming websites) are, er, respectable
3) id Software rather have someone else (in this case, NVIDIA... or, in my opinion, any other IHV) do all the arrangements, something they probably do want to spend the time on doing themselves

After a day mulling over things, however, I decided to write Marty again :

Hi again Marty,

I've thought a bit more about this. Could you let me know why it may not be possible for me/Beyond3D to arrange for a similar benchmarking session directly with id Software? We would really appreciate it and prefer this route than to go through an IHV.

We do have staff based in US that can meet with you/id at id's office for a day if this was possible. Is it possible for one of our staff to meet with you to discuss the possibilities? Do you prefer we discuss this over the phone?

We are willing to meet any NDAs required and are prepared to do this under any stringent security measures you may implement.

If you're a bit apprehensive because you're not familiar with me/Beyond3D, feel free to browse our website or ask me any questions you may have. You may also wish to enquire with ATi (we're a "Tier 1" review website for them) or Futuremark (where we're one of three exclusive press beta members) for references.

Thanks again Marty and hoping for your response.

All of the above, however, isn't as important as the fact that I have not received any acknowledgement from Marty on my last email (the one directly above) after 5 days. Which I take to mean Marty/id is simply not interested. I'm So Sad.

Dave has also done his part in terms of liasing with NVIDIA and the feedback is not good as well. It simply looks as if Beyond3D will not get the chance afforded by those priveleged websites.

<u>The Matrix Reloaded</u>
It rocks. I cannot belive some of the things I saw in the movie. I didn't think it was possible. Guess I simply don't know that much about the technical things behind making movies. I thoroughly enjoyed the movie. It was entertaining (I went "Whoa!" a few times in some of the scenes), which is what I expect of a movie like this. I told my wife "Don't worry if you don't understand what the hell the actors are saying... just enjoy the action scenes"... and I have to say I didn't really catch all that The Architect of The Matrix said in the final scenes. The only disappoinment was that the I didn't get to see the short preview of Revolution because the damn cinema guy cut the end-credits short.

<u>Arsenal... FA Cup champs</u>
I enjoyed their victory but I didn't enjoy the fact that they simply didn't kill of the Saints. Sad to say, this is the main reason they capitulated to the Red Devils in the championship race -- they failed to kill off their opponents in many matches they dominated and had had numerous scoring chances. I also think Ruud van Nistelroy would've gone down (and got an indisputed penalty) if he was Thierry Henry in that first minute of the final where Henry's shirt was so obviously pulled by the Southhampton defender -- Henry is obviously very, very gifted but it is in such split-second decision-making instances where his ego got the better of his mind. He should've gone down and got the penalty. Sums up Arsenal I think.

 

[Joe DeFuria]

Which I take to mean Marty/id is simply not interested. I'm So Sad.

I'm genuinely sad too. (And I thought your follow-up e-mail to Marty was well written and gave them all the information they needed to at least give you a response.)

However, we can also take it to mean not that id isn't interested, but that there is an agreement between id and nVidia, such that going through nVidia is the only way one can get a "benchmark session" on Doom3 (until a predefined period of time elapses, or an event occurs, such as the game going gold). And Marty/id simply doesn't want to admit this, because of how it "looks."

 

[RussSchultz]

I'll have to disagree with your conclusion, Joe. I don't think its conclusive of anything.

But, if you make a big enough stink about it, maybe you'll force somebody to say something more revealing.

 

[pascal]

3DMark03 issue
Well, it is time to have different benchmarks. Maybe some with Random action. Random is not bad as long as you have enough time to have a gaussian distribution of large samples. This is the way the gambling industry works

Doom3 benchmark
IMHO id never really care about benchmarks. I remenber sometime ago B. Hook with simple numbers without much explanation about the numbers and some subjective impressions. My guess they are hard at work to finish this expensive game. I will be worried if id come with something like "the way it is meant to be played" thing.

The Matrix Reloaded. I had no time to see it

 

[Joe DeFuria]

I'll have to disagree with your conclusion, Joe. I don't think its conclusive of anything.

?? I made no conclusions, Russ. Just offered another very logical explanation for the lack of response other than simply "they're not interested, so they're ignoring Rev."

I freely submit either case is a possibility...and personally, I'd weight them about equally possible.

But, if you make a big enough stink about it, maybe you'll force somebody to say something more revealing.

Quite possibly...though it would probably take someone with a "popular web site" to make a big enough stink out of it for id to care. And then I suspect such a web site would not be on id's "good side" (because there is the implicit accusation that id is not being "fair.")

So, it will take some balls for a web-site to try and make a stink out it.

 

[RussSchultz]

Sorry, I must have misread. Your statement came off as "its sad, and we can also take it to mean xyz". That read as a conclusion to me. If not, pardon.

 

[MrsSkywalker]

All of the above, however, isn't as important as the fact that I have not received any acknowledgement from Marty on my last email (the one directly above) after 5 days. Which I take to mean Marty/id is simply not interested. I'm So Sad.

Perhaps they are leary about the level of accuracy the B3D folks expect. It's pretty easy for them to answer the basic info question that Anandtech and HardOCP would be interested in. They tend to try and cater more to the mainstream these days, and while I am not nearly as hardcore geek as my husband, I get very bored reading their articles. I also frequently find grievous errors in them. They still go on and on, though, so this is the level of technical inaccuracy their readers are comfortable with (not knocking those sites, exactly...some people don't give a rat's ass about the finer nuances of anti alaising and such). However, on this board, if they give an off the cuff answer to a question that isn't 100% accurate, they'll be shredded on the boards

Chin up, Reverend...if you look at it in this light, it's rather a compliment

Of course, on the other hand they could think that you're a psycho stalker who won't leave them alone.....

 

[Reverend]

Well, what do you know. Marty just confirmed it while still offering me hope, like he did by saying "We'll keep you in mind" re my request for exclusive Doom3 screenshots a while back :

We're not doing anything like this internally at this point. The
benchmarks last week were a very special case. If we do something in
the future, I'll keep you in mind.

Thanks.
Marty

Let's just hope (again). Of course, fresh exclusive D3 screenshots have since appeared and I didn't get any so maybe Marty is just being polite.

 

[Dave Baumann]

Dave has also done his part in terms of liasing with NVIDIA and the feedback is not good as well. It simply looks as if Beyond3D will not get the chance afforded by those priveleged websites.

Well, it looks to me as though this stems from the fact that the benchmark is not is a 'distributable' format.

 

[pcchen]

My mother stopped me from going to movie theater to watch the matrix because of SARS...

 

[K.I.L.E.R]

Dave said he was very depressed about the whole thing, what it means to Beyond3D (basically there isn't anything we can do to prevent this from happening at the source) and what it means to the consumers. I wouldn't say I'm too depressed but it is worrying because it affects the very nature of Beyond3Ds existence.

Can you please expand on this statement? I just don't see this affecting Beyond3D's existence at all.

 

[Reverend]

My mother stopped me from going to movie theater to watch the matrix because of SARS...

In my previous Pulpit post, I said that I banned cinemas for the family. That was a while back. I asked my wife if we should still be worried about SARS if we are to go to the cinemas and she said "No". In Malaysia, we have 7 official "suspected" cases, with 2 deaths out of that. We went to see Reloaded. My son developed a slight fever *during* the movie and we thought nothing of it. That was two days ago. He is experiencing fever off and on now but they are not high fever and right now the fever is gone.

I cannot tell you how worried and paranoid I got.

 

[Reverend]

Dave said he was very depressed about the whole thing, what it means to Beyond3D (basically there isn't anything we can do to prevent this from happening at the source) and what it means to the consumers. I wouldn't say I'm too depressed but it is worrying because it affects the very nature of Beyond3Ds existence.

Can you please expand on this statement? I just don't see this affecting Beyond3D's existence at all.

Dave has publicly said sometime ago that Beyond3D is a review site. That is what I meant by the word "existence" -- it affects the nature of our site's existence, which is reviewing.

 

[MrsSkywalker]

I just don't see this affecting Beyond3D's existence at all.

Neither do I, but I think I understand the concern. If benchmarks are cheats, than data collected from them can't be trusted at all. If the data can't be trusted, the conclusions in any articles about hardware collected from said data can't be trusted, either...See where I'm going? If there is no reliable information, then there can't possibly be reliable breakdown articles about new hardware...why have a site that is full of misinformation? I think the fear is that this site's credibility will be at stake, and once that happens, membership will wan and eventually the site will be dead. If B3D continues to use 3DMark03, then they will be labled as NVidiots and only taken seriously by NVidia fans. And if they appear to have a bias, then how many other companies are really going to give them products to test? I think they'd be afraid to lose out...if you think someone is already biased against your product, then are you really going to believe they'd give your product a fair shake?

Of course, this is ridiculous. No one knew the bench was Nvidia biased when they first ran the tests. Hell, EVERYONE based their analysis on the results of 3DMark03, and if it is proven that it is a cheat and not a bug or anything else, it does beg the question: What now? Every major bench out there would be under massive scrutiny. If it runs better on Nvidia, does that mean it's another cheat? Or do they just have the better product? It would be a sticky situation for ALL reviewers, not just B3D.

I think B3D has two advantages over some other sites, though: Integrity and an extremely intelligent (for the most part ) membership base. I hope that B3D will be a major factor in the task that reviewers now face: implimenting a new unbiased system of benchmarking. It's a daunting task, but it is necessary. For so long we've all blindly trusted the major benchmarks. We obviously can't do that anymore if 3DMark or Nvidia has been proven to be a cheat. I'll be honest. I have absolutely no idea how to make a benchmark, or any ideas on alternatives (I haven't coded anything since my Basic days...never got into it) But I think that there are plenty of devoted geeks here who do. And I don't think there is one person here who would let B3D slip quietly into the night.

Edit: some typos and jumbled thoughts

 

[pcchen]

In my previous Pulpit post, I said that I banned cinemas for the family. That was a while back. I asked my wife if we should still be worried about SARS if we are to go to the cinemas and she said "No". In Malaysia, we have 7 official "suspected" cases, with 2 deaths out of that. We went to see Reloaded. My son developed a slight fever *during* the movie and we thought nothing of it. That was two days ago. He is experiencing fever off and on now but they are not high fever and right now the fever is gone.

I cannot tell you how worried and paranoid I got.

Currently SARS is spreading fast in Taiwan, and everyone is anxious about it. So I guess I won't go watching movies for quite some time I bought a Logitech Formula GP steering wheel for my PS2 to "compensate" the boredom in weekend.