GeforceFX - what will be the best way to benchmark it?

[g__day]

Personally I think this will be a very complex card to analyse, like nothing else that has gone before it, ever.

You have to look at it vs the GeForce 4 Ti 4600 and Radeon 9700 Pro, you have to look at it under Directx 8.1 and Directx 9 (due Dec 10th I hear though a only sometimes reliable grapevine).

The Radeon 9700 Pro is a different card to the GeForceFX (ugh - what a mouthful). The 9700 Pro is meant to add a real boost for today's games plus be DX 9 ready and able. The GFX is I believe positioned more squarely at DX9 and beyond. I wonder that NVidia may be swapping positions with ATi - going for elegance over brute strength to win.

I think one must be EXTREMELY careful in deciding how to benchmark these cards to give a balanced review. With CineFX NVidia is targeting not just the gaming but the film industry too. Maybe not today - but very soon. They have very different engine rooms in their respective GPUs - you need both old software and new utilities to do apples versus apples comparisions. I bet ATi and NVidia will release interesting demos suited to their own beast and showing the opposition in a poorer light.

I want to see what John Carmack says about this card in his next .plan update - not his 1 year old comment.

I really want to see benchmarks with alot of thought behind them. Alot of the raw power behind NV30 will be how good LMA 3 is to let 16GB/sec compete against 19GB/sec. Somehow I think NV30 will hit very, very much above its weight here IF used well...

* * *

What are peoples thoughts about the best way to analyse GFX and 9700 - do we need something special, do we have the toolset tpo really dissect and analyse both chips? Are all the old approaches still the best way?

 

[Fuz]

Personally I think this will be a very complex card to analyse, like nothing else that has gone before it, ever.

Considering that its very similar to the R300, I can't see why you would think that?

The Radeon 9700 Pro is a different card to the GeForceFX (ugh - what a mouthful). The 9700 Pro is meant to add a real boost for today's games plus be DX 9 ready and able. The GFX is I believe positioned more squarely at DX9 and beyond. I wonder that NVidia may be swapping positions with ATi - going for elegance over brute strength to win.

Seeing that DX9 games won't appear for atleats 18 montsh (mho), I doubth they would release a card aimed for DX9 and beyond. Maybe the NV40 and beyond, but not the NV30.

I think one must be EXTREMELY careful in deciding how to benchmark these cards to give a balanced review.

I agree, as cards get more adjustable, as far as image quality goes, reviewers need to take care to make sure they are comparing apples to apples, not oranges.

What are peoples thoughts about the best way to analyse GFX and 9700 - do we need something special, do we have the toolset tpo really dissect and analyse both chips? Are all the old approaches still the best way?

IMHO, old approaches are fine, as long we keep in mind its all about image quality now. Don't really care weather its one companies 8x AF vs. another companies 512x AF ( ) as long as final out put (image on my damn monitor) is comparable.

Of course, its all jmho

 

[demalion]

Hmm...I don't quite buy the "brute force versus elegance" comparison between R300 and NV30, as it sounds more like hype rather than what the actual products seem to be. I really don't think the cards are very much different at all in any regards to their benchmarking demands.

That said, I don't think we have the tools to expose their relative strengths.

Rightmark 3D (or is it Rightmark Video Analyzer?) listed DX 9 tests as a target for its final functionality...I've seen versions used in benchmarks, but I don't think it is the final product (my first clue is that DX 9 isn't shipping ). Perhaps that will offer some insights.

Also, those "pretty" games nvidia mentioned with the nv30...are they using floating point precision, or are they just using DX 8.1 functionality to achieve their "prettiness"? I wasn't there, and I can't tell from the small screenshots I've seen so far. My own theory is that DX 8.x -> DX 9 is a pretty small step in terms of time investment for immediate rewards (floating point precision processing) so I am hoping some or all of them mentioned by nVidia are going to expose some functionality of this type.

In any case, the only important thing I see for current games is image quality comparison in detail and performance analysis where demonstrably equivalent image quality is present. And this is for high image quality, as that is all that matters for the respective high end cards (9700 and whatever is the "high end" nv30), and not base. What I EXPECT, on the other hand, is a hodge podge of sloppy comparisons that will leave me frustrated and badgering the Beyond3D staff to hurry up and stop sleeping for a week or two and get me the benchmarks and comparisons I want... (sorry in advance ).

I wonder if Humus could come up with some stuff, though? Might be educational for him to turn his considerable talent and focus it on metrics of this type in anticipation of the nv30 launch, given the type of career path he is taking...the experience could boost his paycheck a bit, methinks...

 

[jjayb]

I see Nvidia marketing is working. Making people think this card deservers it's own type of benchmark, when it's not so different from the r300.

 

[T2k]

I see Nvidia marketing is working. Making people think this card deservers it's own type of benchmark, when it's not so different from the r300.

Well said...

 

[WaltC]

Personally I think this will be a very complex card to analyse, like nothing else that has gone before it, ever.

You have to look at it vs the GeForce 4 Ti 4600 and Radeon 9700 Pro, you have to look at it under Directx 8.1 and Directx 9 (due Dec 10th I hear though a only sometimes reliable grapevine).

The Radeon 9700 Pro is a different card to the GeForceFX (ugh - what a mouthful). The 9700 Pro is meant to add a real boost for today's games plus be DX 9 ready and able. The GFX is I believe positioned more squarely at DX9 and beyond. I wonder that NVidia may be swapping positions with ATi - going for elegance over brute strength to win.

What makes you say something like that? The nv30 and R300 both do 8 pixels per clock. The nv30 has only slightly less bandwidth than the 9700 Pro. However, nv30 is using 1GHz DDRII ram atop a 128-bit bus, whereas ATI is using a 256-bit wide bus and running DDR1 @620MHz--of the two I consider the nv30's to be more "brute force," actually. Additionally, it would appear from circulated images that the nv30 chip is being appreciably overclocked by the size of its monstrous heatsink--which the 9700 Pros do not come close to copying. Again, nv30 proves more of a "brute force" solution there, IMO.

So, with more theoretical pixel fill rate and a higher GPU clockspeed, and only a slightly lower bandwidth, just how is nv30 *less positioned* to run "today's games" than the 9700 Pro?

Both GPUs have DX9-capability--which means they can both run DX9 software, and so how is the nv30 better positioned for them? Your comments sound as if you read them somewhere in a speculative article--you might want to consider them a bit more. In terms of Pixel-pushing power, the nv30 clearly has more "brute force" than the 9700 Pro, although that is mitigated somewhat by not having as much bandwidth as the 9700 Pro. Elegance, and so forth, is an opinion.

I think one must be EXTREMELY careful in deciding how to benchmark these cards to give a balanced review. With CineFX NVidia is targeting not just the gaming but the film industry too. Maybe not today - but very soon. They have very different engine rooms in their respective GPUs - you need both old software and new utilities to do apples versus apples comparisions. I bet ATi and NVidia will release interesting demos suited to their own beast and showing the opposition in a poorer light.

The largest market for High-end 3D cards right now, by far, is 3D gaming. Versions of nv30 made by various OEMs for the 3D modeling crowd (eg, Quadro, et al) will ship with a lot of high-end, professional software, and cost a lot more than the consumer versions, which will serve the 3D gaming market. I think benchmarking with actual 3D games is appropriate for both. Demos, of course, show off a product, and really are a minor consideration.

I want to see what John Carmack says about this card in his next .plan update - not his 1 year old comment.

I'm sure that when he gets one (if he not doesn't already have a prototype) he'll have plenty to say. But when showing off the DIII demo what he said of the 9700 was that at the time it was the only 3D card that could handle it. That will obviously change once the nv30 gets out and around.

I really want to see benchmarks with alot of thought behind them. Alot of the raw power behind NV30 will be how good LMA 3 is to let 16GB/sec compete against 19GB/sec. Somehow I think NV30 will hit very, very much above its weight here IF used well...

What gives you that idea? Smarter not to preconceive notions as to facts not in evidence. That way you can avoid disappointment, or else be pleasantly surprised. Better to accept the bandwidth of the nv30 as what it is--16gigs/sec--for the time being.

What are peoples thoughts about the best way to analyse GFX and 9700 - do we need something special, do we have the toolset tpo really dissect and analyse both chips? Are all the old approaches still the best way?

I think the best way will be with DX9 tools and games when we get them, not to mention OpenGL 2.0 when it becomes a factor. Games based on older technology are fine, too, as many still play them. But it will be DX9 performance, I think, which will be most interesting to evaluate.

 

[Entropy]

I think the best way will be with DX9 tools and games when we get them, not to mention OpenGL 2.0 when it becomes a factor. Games based on older technology are fine, too, as many still play them.

LOL!
Yeah, damn those slow adopters.

It all depends on your outlook. Do you buy a card for its sexy tech or for the job you need done? 99% of B3D posters seem to favour the first, but for some reason can't admit that, and then bitch and moan about the lack of high tech games. It's time you guys to look in a mirror, recognize yourself for what you are, and come out of the closet. Techno lust is OK. Seriously.


Entropy
(Emphasis added by me)

 

[Nagorak]

Testing method: Run current games and compare results. Ignore 3DHype2003 and whatever new gaming engines might be available in the future. People buy cards for the moment, not the future, just compare what's out there and let the numbers speak for themselves: nothing complex about it.

 

[Reverend]

I can't speak for anyone but myself/VE but I'll continue to use popular and/or quite-new games that I can either buy or download and exploit the OOTB 3D features that a gamer can use.

For a review, there isn't any worry about "apples to apples".

For a shootout, obviously final-but-ultimately-subjective screen output should be the determining factor in which card is better/best.

For a 3D-technology-centric site (or one that claims to be), a smattering of available games as well as tech-specific benchmarks are always welcomed. For this, it would be best for such sites to come out with their own form of benchmarks. Here's an example of what one of the regular contributors here informed me he would do (if he ever gets around to it ) for a pure VS benchmark :

...and I think the
first program I'd like to try is a program that tests
the efficiency/performance of the vertex (and later
pixel) shader pipeline by benchmarking every single
instruction and timing how many cycles it takes to
complete.

Roughly, for each vertex shader op, I will generate
shaders that look like this:

----
248 repetitions of instruction "X"
----
basic transform, output position
----

I will arrange the dependencies between such
instructions so that they cannot be "optimized away"
by the driver. For example, for the "add" instruction,
I can simply do a fibonacci-like sequence, and make
the 3rd add instruction dependent on the previous two.

I will then render something simple (at a low
resolution), like a dense rectangular grid of several
thousand vertices. After computing how many vertices
we rendered per second, and subtracting off the known
time to do the basic transform, we will have the time
used by the 248 repetitions of instruction "X", and
then just divide by 248 to get the time used per
instruction.

Then, we can calculate the instruction timings by
dividing that by the time of a cycle clock cycle (say,
300Mhz)

e.g instruction timing for X = ( verts/second divided
by # verts ( gives time for one vertex shader to
complete ) - basic transform timne ) divided by 248
divided by (1 / 300Mhz) = #cycles needed by "X"

After we get that working, we can then benchmark
things like parallel execution of different types of
ops (pairup a scalar, vector, and floating point
divide or SIN/COS function)

We can then use the same technique to benchmark the
real world performance of the pixel shaders. And, we
can benchmark how much a performance hit comes from
dynamic predicates/branching.

Moreover, we can bench the various implementations of
high level instructions like "lit", "exp", "sin",
"cos" to see if they are really implemented by the
hardware, or transformed by the driver into a power
series approximation.

While not exactly relevant to how a piece of hardware may be used in 90% of software (like games) that gets sold reasonably well, it would be interesting hardware-wise.

BTW, when is DOOM3 out? Coz that's the big one, ain't it? If I'm not mistaken, it would be around the retail-availability of the GeForce FX

 

[g__day]

Guys a clarification - I see both the 9700 and GFX worthy of benchmarks well suited to show their full potential. I am not convinced those benchmarks exist today.

I think (hope is more like it) that the NV30 will be effective with its 16GB/sec. I am sure NVidia could have gone down the 9700 path but choose the path they did. Whether this is a wise choice or not we will see.

I agree most testing will be on what it can do today for comparable image quality levels. But I respect these cards are designed to transition us into DX9, it would be great if reviewers absorbed that dimension into their benchmarking.

I feel that by the time DX9 games are common R350 and NV35 or is it GeForce 2 FX will be out and about. I don't feel the immediate need to upgrade.

I hear rumours amongst reviewers that 1) NV30 isn't at full speed yet - hence no demos and 2) the GPU may scale from 500 - 700MHz (probably if you live in Alaska right?)

 

[martrox]

Rev, where's that Radeon 9700 Pro review??????

(sorry, off topic)

 

[Simon F]

GeforceFX - what will be the best way to benchmark it?

Judging from the pictures, perhaps with an ammeter or a thermometer?

I will arrange the dependencies between such
instructions so that they cannot be "optimized away"
by the driver. For example, for the "add" instruction,
I can simply do a fibonacci-like sequence, and make
the 3rd add instruction dependent on the previous two.

That's an interesting idea but the results would depend very heavily on the latency of the instructions whereas the throughput may be considerably higher. I think the 9700, for example, appears to have a latency of about 3 or 4 instructions.

 

[Humus]

I wonder if Humus could come up with some stuff, though? Might be educational for him to turn his considerable talent and focus it on metrics of this type in anticipation of the nv30 launch, given the type of career path he is taking...the experience could boost his paycheck a bit, methinks...

I suppose I could write a shader benchmark. One could use several different shaders with different characteristics, like one shader using mostly "basic" math, like add/mul, another using loads of rcq/rcp/exp/log stuff, and some shader which little instruction dependencies and another with a lot of dependencies, etc.

 

[Reverend]

PM to martrox... carry on.

 

[Entropy]

There are basically two styles of benchmarking.

One which uses applications or application-like codes in order to be able to predict directly how well different systems will perform at a particular task or class of tasks. Example: Q3, mapping to 3D-game performance.
Main problem: Transferability of results, i.e. the benchmark doesn't model the target sufficiently well that you can draw useful conclusions from the data. The underlying reasons for a given score are hidden and can be misattributed.

The other which measures performance of well defined subsystems. Various contributing factors to performance are explicitly exposed. If you know what is important for your particular application, you can predict relative performances by appropriately weighing results from only those subsystem tests that are relevant. Example: 3DMark2001 subsystem tests, mapping to 3D-game performance.
Main problem: It takes considerable knowledge to go from subsystem performance data to an accurate application performance prediction, often impossible without knowing the source code. If there are additional factors that are relevant for application performance that are not included in your data (Example:host system performance) then making useful predictions are, again, nigh on impossible.

If you are proposing the creation of a new benchmark for new generation hardware, it is very important to know exactly what you want to test and why.

And it should always be remembered that benchmarking with the precision necessary for generated data to actually be useful is not necessarily possible at all, unless the target set of problems is extremely narrow.

Incidentally, a year or so ago, I made an informal, now somewhat outdated, statistical analysis correlating the 3DMark2001 subsystem tests with application performance. Application performance almost exclusively depended on two factors - pixel fillrate (major) and texture fillrate (minor). No other factor contributed enough to register significance. That is gfx-card fillrates and host performance was sufficient to predict how games would run. (Actually, better information could be had from running Q3 at very low and very high resolutions, since that gives both host performance and gfx-card data, but includes the effects of for instance occlusion culling that doesn't show up in 3DMarks subsystem tests.) All those graphs filling webpages basically said the same thing, over and over only with different balances of host vs. gfx-card performance.

I don't know if in the future there will be any single parameter that stand out as performance critical the way fillrate has done up to now. If not, the outlook for benchmarks to actually have much predictive value is grim.

Entropy

 

[KimB]

Personally, I think that NV30 benchmarks should take a few different forms.

First, benchmarks in current games are indispensable. This is a gaming card, and its performance in today's games cannot be ignored.

Benchmarks should definitely focus on high-quality modes (FSAA, anisotropic filtering).

But, to get a more full view of what the video card is capable of, benchmarks that analyze the advanced features are also important, though they should mean less for the average user.

 

[WaltC]

I think the best way will be with DX9 tools and games when we get them, not to mention OpenGL 2.0 when it becomes a factor. Games based on older technology are fine, too, as many still play them.

LOL!
Yeah, damn those slow adopters.

It all depends on your outlook. Do you buy a card for its sexy tech or for the job you need done? 99% of B3D posters seem to favour the first, but for some reason can't admit that, and then bitch and moan about the lack of high tech games. It's time you guys to look in a mirror, recognize yourself for what you are, and come out of the closet. Techno lust is OK. Seriously.


Entropy
(Emphasis added by me)

Yea, the "as many still play them" was a pretty flat-footed way of stating the obvious, wasn't it?... What else is there to play, right now? *chuckle*

 

[Joe DeFuria]

Benchmarks should definitely focus on high-quality modes (FSAA, anisotropic filtering).

Yes, and for God's sake, there should be at least SOME attempt (when doing "shoot-outs") to make the preformance comparisons as directly apples to apples as possible....by observing the relative image quality.

 

[WaltC]

g_day,

This is where I don't quite follow:

I am sure NVidia could have gone down the 9700 path but choose the path they did.

What's the difference in nVidia's path?

 

[BRiT]

I am sure NVidia could have gone down the 9700 path but choose the path they did.

What's the difference in nVidia's path?

Nvidia's path seems to take several months longer and quite a few degrees hotter.

For now, there is no way (for web-reviewers) to benchmark the NV30 card. It may as well still be nonexistent/vaporware. Most disappointing launch ever.

--|BRiT|