Neutral Benchmarks

[AJ]

Just finished reading Lars editorial at Tom's Hardware (he phoned us up after the first write up. Lars, if you're local to the forums here a big thanks for listening to us too!) and I think that's also a very good and fair assement of the situation.

With regard to the FPS vs. 3DMark score: By all means _do_ use the FPS's for your reviews! This is what for example Anand does and I think that's a very smart move. It gives you as the news editor the possibilty to do your own balancing or weighting as you please and we have absolutely nothing against it.

The reason why we sum it all up to a single number is to make things easy for those who don't want to bother. This makes comparison for 'regular joe sixpack' a lot easier.

Cheers,

AJ

 

[Hellbinder]

Personally, I dont see any difference between a point total, and an actual FPS. Especially since the actual FPS that coinside with the score are available to all and very easy to see.

Interesting note on the lars update. The only real irkish thing in his editorial for me was is comments on PS 1.4
http://www.tomshardware.com/column/20030219/3dmark2003-04.html

When forced to use PS 1.1, the Radeon 9700 Pro takes quite a performance hit compared to PS 1.4. It also seems to support NVIDIA's cry of foul play - at least at first. Changing perspective once again and looking at what 3DMark 2003 is meant to be, namely a platform to evaluate native, unoptimized DirectX 9 performance of graphics cards, the use of PS1.4 makes a good deal of sense. After all, as stated above, all DirectX 9 cards (have to) support PS1.4. There's even a handful of DirectX 8.1 cards (i.e., the Radeon 8500/ 9000 lines) that support it, making it a very attractive option for game developers. Or, as Futuremark puts it:

Its not that this is a bad comment. Its simply that he seems to Fail to understand that the reason the scores drop off 1,000 points when forcing PS 1.4 is the 9700 can no longer run game 4. It gets a big fat "0". Which nixed 1/4 of the score. It makes perfect sense. When you take the next step and Force PS 1.1 the cscore is only affeected by an additional 400 points. Which is the actual weight you should give to the impact of ps 1.4 when compared to the GF4 line. 400 points does not mean a whole hell of a lot.

So while he does give his conclusion a posotive spin. It irritates me to no end when people can not grasp the simple data in front of their face.

That is all the negative commentary i have for lars. He has definitely come a long way. I wonder if kyle/Brent will see fit to offer a similar *corrective* editorial. I personally dont see it hapening. From the personal exchanges i have had with Kyle over this, I can tell he is dead set. Hook line and sinker. You could show him a roswell alien in the flesh and he would still reject it...

 

[mr]

First thanks Lars for your new column. As several people already mentioned this is much better and written in a fair way using information from several parties involved in the discussion, while making very clear distinctions between your own opinion and the opinion of the other parties involved.

Its not that this is a bad comment. Its simply that he seems to Fail to understand that the reason the scores drop off 1,000 points when forcing PS 1.4 is the 9700 can no longer run game 4. It gets a big fat "0". Which nixed 1/4 of the score. It makes perfect sense. When you take the next step and Force PS 1.1 the cscore is only affeected by an additional 400 points. Which is the actual weight you should give to the impact of ps 1.4 when compared to the GF4 line. 400 points does not mean a whole hell of a lot.

He could have worded it clearer, but IMO he talks about the performance difference of GT2 and GT3 when using PS1.1 compared to PS1.4, changing fps from 30.6 to 24.9 for GT2 and from 28.2 to 22.9 for GT3 which could be attributed as "quite a performance hit" (about 20%).
I certainly agree with you that the overall score dropped only 400 points, which is IMO not that bad (about 12%) as it includes GT1 which (unsurprisingly) did not change at all.

 

[Evildeus]

Its not that big!

Then why did you, and now THG, in the first instance spoke of a "big list"? Seems like Ati marketing is working well also in some places.

I really don't understand: Why don't you publish this list? Please explain....

*edit* ROFLMAO 6 games and 1 not even finish... I understand the "that big" now

UT2003
Madden 2003
Tiger Woods 2003
Nascar 2003
NeverWinter Nights (actually OGL equivalent).
DOOM3 will have a path that will use the equivalent of PS1.4 as well.

 

[Dave Baumann]

Well, the one I have is bigger than that. But, patience!

 

[Evildeus]

Well, the one I have is bigger than that. But, patience!

Could once for all explain why you don't publish this list? Actually, that's make 2/3 days you are talking about it, but nothing appears. I'm wondering why

 

[Mariner]

Patience?

Seems like the 'long list' must be part of a forthcoming preview/review of something?

 

[Ante P]

Joe, check your PM. I thought we could continue the discussion there instead
else we'll just post this thread to death it seems

 

[jb]

AJ

part of the reason why I think no scoring would be better is that it removes the encentive to optimise only for the test with the highest weights. It makes each and every test important and I think thats better for joe six pack then making sure GT3 for example runs faster just because it has a higher weight than GT1.

User freindly? Well to be honest people now don"t seem to understand what the score means so why give it to them?

Joe

Sorry I dissagree.

What does 24 AVG FPS tell you for GT3? That the card that does that can only run DX8 games at 24 FPS

So what does teh final 3dmark2k3 score of GFX vrs R300 tell you? Not much really. Does it highlight the differences of how the cards run future DX8 games? What about DX9 games? Its very easy that one could do better in a DX8 game and suck in a DX9 game and yet it could recieve a higher 3Dmarks2k3 score. You really have no idea unitl you look at each test and say ahhhh one does better in GT2/3 but not so well in GT4. Thats usefull. X marks is worthless.

I understand how some sights can screw it up. But these sights are screwed up now about what 3dmark scores mean

 

[nelg]

Well said Lars ! Reading this has helped restore my opinion of "THG". I feel that DeanoC summed up best why Nvidia may have a problem with 3DMK03....

Do you think that this stems from vNidia being a bit, well, twitchy about whether they wanted to target 'ordinary' consumers with int and FP16 paths vs FP32 for the prof/dev market with the FX architecture?

I think its much simplier - performance, they made a bet that 32 bit float aren't needed much so it was o.k. for them to run much slower (50% of the speed).

Along came ATI and produce 24 bit floats, which have the best of both worlds - as fast as 16 bit floats and have the precision to be really useful. Because they were first (by a big margin) they effectively set the minimum spec.

Which leaves them trying desperate (and in this case stupid) things to get some performance back.

There is no reason for them to force DirectX devs to use 16 bit floats! We can specify partial precision, so they just have to educate us to use it when we can.

I think that sums things up.

 

[BoddoZerg]

Why does everyone focus on the NV30/R300 in their 3dmark arguments? It seems obvious to me that the reason nVidia doesn't like 3dmark03 is because of their GeForce4 line... both GF4MX and GF4Ti score abysmally compared to similar-priced Radeons.

I wouldn't be surprised at all if, by next year, with NV30/31/33/34/35 roughly comparable in 3dmark03 against R(V)300/350, nVidia pulls a 180 and tells everyone to use 3dmark.

Then again, maybe I'm just crazy.

 

[nelg]

I am begining to think Futuremark should combine 3DMK01 with 3DMK03 so when started the program can decide which one to execute. My current computer has a Gefore MX400 and it would serve no purpose to run 3DMK03, or 3DMK01 for that matter .

 

[Hellbinder]

edit* ROFLMAO 6 games and 1 not even finish... I understand the "that big" now

Tell me, how many selling titles use PS 1.2 ore 1.3???

Again, If there are say 10 AAA classed major titles of the type that would use such technology a year. And PS 1.4 is used in at least half of them. I dont see the problem. It will have become a very valid technology. especially if the support is built into the engine.

All games Using the Doom-III engine
All games Using the Unreal Engine
Various comming DX8 titles

Just the top Two alone will account for at least 7 MAJOR games comming out that will sell hundreds of thousands of copies.

It just seems like when we start talking about games and feature support the *games* become this ambiguous term that gets thrown around. Like there are hundreds of games a year worth even buying. Thats just not the case. There are an average of about 5 Games a month released(at least games that anyone even knows about). Only a handfull of them will even use any kind of PS technology at all. If PS 1.4 gets used in several of the Key titles each year.

How can anyone make a case against it being a valid feature?

 

[Joe DeFuria]

part of the reason why I think no scoring would be better is that it removes the encentive to optimise only for the test with the highest weights. It makes each and every test important and I think thats better for joe six pack then making sure GT3 for example runs faster just because it has a higher weight than GT1.

3 of the 4 tests are weighted the same at 26.7%, and one test (GT4) is weighted at 20%. I don't follow you....

User freindly? Well to be honest people now don"t seem to understand what the score means so why give it to them?

You can lead a horse to water, but can't make him drink. Doesn't mean you shouldn't lead him to water.

Joe

Sorry I dissagree.

Heh...on what? Everything?

So what does teh final 3dmark2k3 score of GFX vrs R300 tell you?

If we go by what FutureMark says, (and assuming the latest drivers for the FX are legit) it means that the GFX Ultra and Radeon 9700 Pro have very similar "playability" with respect to next-gen games coming in the next 6-18 months.

at least Carmack seems to agree with that assesment, as would most people on the board given the little information we have at this time.

Its very easy that one could do better in a DX8 game and suck in a DX9 game and yet it could recieve a higher 3Dmarks2k3 score.

Possibly, yes. What's your point though? If more games next year stress DX8 level features as opposed to DX9, wouldn't the higher 3D Mark score for the card that sucked in DX9 be a better representation of "overall playability"?

You really have no idea unitl you look at each test and say ahhhh one does better in GT2/3 but not so well in GT4. Thats usefull. X marks is worthless.

No one is saying you shouldn't look at and examine individual scores and individual feature tests for that matter. You can get ADDITIONAL information from them.

If you think that the game tests are OK, and just don't like the SCORE, then make up your scoring formula if that makes you happy, such that it comes up with a number that "quantitatively" represents, overall, how good you think each card is, relative to one another.

The point is, do you think OEMS even WANT to do this? They want "a socre." Consumers looking at boxes want A SCORE. Do you think 3DMark's current weighting scheme is so far out of line that it's just flat out unreasonable?

I'm sure we'd all agree that ANY "single number" can't possibly do card comparisons complete justice. But the question is: from a practical standpoint, as a single number, is 3D Mark '03 leading anyone in the wrong direction?

 

[Crusher]

]400 points does not mean a whole hell of a lot.

That's 13 times my total score, it must mean something.


As far as PS1.4 goes, arguing about it's usefulness, market saturation, etc. is a bit of a moot point. There really aren't that many cards that only support up to PS1.4 in use, certainly not nearly as many as those that support lower versions or none at all. And ultimately it doesn't matter, because the cards that do support it really aren't going to run games any faster than the cards that don't. That is, a game that uses PS1.4 but falls back to PS1.1-1.3 probably isn't going to run slower on a GeForce Ti4600 than it is on a Radeon 8500 because of it (indeed, I'd like to see benchmarks of any game in that list where it really makes a difference in terms of performance). In fact, last we knew, Carmack was saying the Ti4600 was running faster than the 8500 despite having to do more passes.

By the time there are games that use extensive pixel shaders versions 1.4 or higher, you'll need a better card than a Radeon 8500 to run them acceptably anyway. That pretty much means you'll need a DX9 level card to run any high-tech PS1.4 game, and since all DX9 cards will support PS1.4, nobody should be complaining. Except there is still the problem that those types of games are at least 2 years away, and 3DMark03 is here right now, influencing purchasing decisions. That's good for the game industry, as it helps increase the demand for DX9 level hardware. It's bad for NVIDIA, because OEMs might choose to go with ATI instead because they get better 3DMark03 scores. None of this has anything to do with the fact that 3DMark03 isn't representative of what games are like now, or will be like in the future, or how cards will actually perform in either, apart from simply showing that something does or does not support certain features..

 

[Evildeus]

]
Tell me, how many selling titles use PS 1.2 ore 1.3???

What's the point? If you don't see why it's humoristic, i can't do anything for you.

As you say, "gimme a break"

 

[Joe DeFuria]

That is, a game that uses PS1.4 but falls back to PS1.1-1.3 probably isn't going to run slower on a GeForce Ti4600 than it is on a Radeon 8500 because of it (indeed, I'd like to see benchmarks of any game in that list where it really makes a difference in terms of performance).

That's not the point.

A game that uses PS 1.4 with a fall back to PS 1.1-1.3 will run better on a Radeon 8500 than a game running on 1.1-1.3 only.

It's bad for NVIDIA, because OEMs might choose to go with ATI instead because they get better 3DMark03 scores.

It's bad for nVidia indeed. Having a lower 3DMark score is a consequence of having an inferior product. Right.

 

[Hellbinder]

As far as PS1.4 goes, arguing about it's usefulness, market saturation, etc. is a bit of a moot point. There really aren't that many cards that only support up to PS1.4 in use, certainly not nearly as many as those that support lower versions or none at all. And ultimately it doesn't matter, because the cards that do support it really aren't going to run games any faster than the cards that don't. That is, a game that uses PS1.4 but falls back to PS1.1-1.3 probably isn't going to run slower on a GeForce Ti4600 than it is on a Radeon 8500 because of it (indeed, I'd like to see benchmarks of any game in that list where it really makes a difference in terms of performance). In fact, last we knew, Carmack was saying the Ti4600 was running faster than the 8500 despite having to do more passes.

You have got to be kidding me... The 8500 is only one small part of the puzzle. All the R300/R350/Rv350 based cards are not only DX9, but every bit as much DX8.1 cards. All supporting PS 1.4

The 8500 has some other hardware disadvantages that make it slower than the GF4 in doom-III. But that is only about 1% of the argument as I have already demonstarated above. And NO I do not agree with your assesment of PS 1.3/PS 1.4

There are one hell of a lot more DX 8 related games coming out long before primarily DX9 games hit the shelves anywhere.

By the time there are games that use extensive pixel shaders versions 1.4 or higher, you'll need a better card than a Radeon 8500 to run them acceptably anyway. That pretty much means you'll need a DX9 level card to run any high-tech PS1.4 game, and since all DX9 cards will support PS1.4, nobody should be complaining. Except there is still the problem that those types of games are at least 2 years away, and 3DMark03 is here right now, influencing purchasing decisions. That's good for the game industry, as it helps increase the demand for DX9 level hardware. It's bad for NVIDIA, because OEMs might choose to go with ATI instead because they get better 3DMark03 scores. None of this has anything to do with the fact that 3DMark03 isn't representative of what games are like now, or will be like in the future, or how cards will actually perform in either, apart from simply showing that something does or does not support certain features..

I already answered this. I just dont understand your fixation on the 8500. You are tryiing to use 1 card out of 6-8 other PS 1.4 cards to justify your arguments. 3dmark03 shows how Ps 1.4 benefits Dx8 far more than it has any indication of DX9.

 

[Hellbinder]

What's the point? If you don't see why it's humoristic, i can't do anything for you.

As you say, "gimme a break"

Being that your list is not correct. And being that i made a pretty clear case on what we are really talking about when we say *games*.. is termas of type and number...

If your list was correct, then sure it would be funny.

 

[John Reynolds]

]I already answered this. I just dont understand your fixation on the 8500. You are tryiing to use 1 card out of 6-8 other PS 1.4 cards to justify your arguments. 3dmark03 shows how Ps 1.4 benefits Dx8 far more than it has any indication of DX9.

Problem with this logic is that Nvidia probably still dominates developers in terms of being the primary hardware platform. I can't quantify this, but I'm probably right.