Chris Egerter on the priveleges of Futuremark

[Reverend]

Chris' latest update

I find it interesting that Futuremark works with ATI and Nvidia to make the benchmark run as fast as possible on specific hardware. Game developers don't have the luxury of doing this. Publishers want them to finish the game as quick as possible and most of them don't have months to spend tweaking code paths for specific cards, especially when new cards are coming out every few months.

Once you have a proven good track record in game development, you'll probably have the same luxury, I guess

I'm sure Carmack and Sweeney enjoys it. In fact, for them, it's probably crucial.

 

[Chris Egerter]

I said most.

That's 2. How about the other thousands of developers that aren't making the next AAA title?

I think we would be better off if Futuremark didn't work with hardware manufacturers, and were completely unbiased. Otherwise we might as well have benchmarks written by ATI and Nvidia tailored to show off specific features on their cards and make the other look bad.

 

[GraphixViolence]

That doesn't seem too realistic. Since 3DMark aims to be a forward-looking benchmark, they have to implement the latest graphical techniques and stress the latest hardware. And they have to do this before games using those techniques become widespread, otherwise people will just benchmark using the games instead.

Regardless of how skilled Futuremark's programmers are, I have to think it would be almost impossible for them to develop a meaningful benchmark without access to pre-release hardware and intensive developer support from ATI and NVidia. In order to avoid bias, they have to try to get support from both companies, as well as other key industry players like Dell, Intel, Microsoft, and even major hardware review web sites.

 

[ET]

From what I understood from our CEO, ATI and NVIDIA are willing to help developers write code that takes advantage of the latest hardware (not that I take what he says as gospel). I haven't tried to get such support, so I don't know what help they provide exactly, but I do know that the graphics companies have interest in games that need the latest hardware, and would support them. I also know that ATI and NVIDIA answer questions relating to performance considerations, because I've asked such questions.

I don't think that working with hardware companies makes a benchmark biased. I think that developers should work with them, on both games and benchmarks. I also think it's natural that companies that don't want to work with developers will suffer. For example, I tried to contact SiS but didn't get a reply, so if the game doesn't run well on a Xabre, I'll see it as partly their fault.

 

[Dave H]

Also it's worth pointing out that all the other consumer 3d IHVs are on the beta program too (or at least all I can think of: Intel, SiS, S3, Matrox, PowerVR, 3dLabs and Trident), although not at the "strategic" level where ATI is and Nvidia was.

It does seem a tad unseemly to have all the vested interests directing the course of where the benchmark goes, but at least it's all in the open, they can all comment on each other's suggestions, and that's the way most other industry-standard benchmarks work (e.g. SPEC, TPC, etc.).