No DX12 Software is Suitable for Benchmarking *spawn*

[Deleted member 13524]

Not sure if you're being ironic, but one 3dmark version (vantage IIRC?) used to have a physx test that contributed to the main score, where nvidia systems would use the GPU and all others had to resort to that old very unoptimized x87 physx code in the CPU.

No, you are not remembering correctly. Sorry. There are, however, 3DMarks that do use higher than base feature levels of DirectX to optimize the rendering process. I await a collective call for justice now...

Turns out I am:
https://www.techpowerup.com/66327/gpu-physx-doesnt-get-you-to-3dmark-vantage-hall-of-fame-anymore
When 3dmark Vantage released in April 2008, the "CPU Physics" test was using PhysX, which was GPU-accelerated if nvidia cards were detected. The total score was affected by a CPU test that was running on the GPU, giving nvidia cards a rather big advantage (mainly because PhysX ran like crap on CPUs at the time).

Futuremark only started to filter out GPU-physics results in late July, after a huge backlash from enthusiasts and, of course, after all the reviews comparing GTX 270/280 and HD4870/HD4850 (which also released in April 2008 BTW) had been published.
The damage had been done among enthusiasts, most decent reviewers saw through it and 3dmark itself was left out of reviews for years to come.


Futuremark may be 100% innocent with this async fiasco, but don't doubt that one of the reasons they're taking shit right now is because they have a criminal record and the Internet, like the North, never forgets.

About physx: that's not a Futuremark fault if the CPU-software version of physx had legacy x87 calls.

It was not Futuremark's choice to evaluate CPU performance using a proprietary physics engine that didn't run on the CPU when GPUs from a single vendor were detected?
Whose choice was it then?

 

[Ike Turner]

Well, that escalated quickly

What a time to be alive. Interesting times ahead..

 

[surreal]

Turns out I am:
https://www.techpowerup.com/66327/gpu-physx-doesnt-get-you-to-3dmark-vantage-hall-of-fame-anymore
When 3dmark Vantage released in April 2008, the "CPU Physics" test was using PhysX, which was GPU-accelerated if nvidia cards were detected. The total score was affected by a CPU test that was running on the GPU, giving nvidia cards a rather big advantage (mainly because PhysX ran like crap on CPUs at the time).
Futuremark only started to filter out GPU-physics results in late July, after a huge backlash from enthusiasts and, of course, after all the reviews comparing GTX 270/280 and HD4870/HD4850 (which also released in April 2008 BTW) had been published.
The damage had been done among enthusiasts, most decent reviewers saw through it and 3dmark itself was left out of reviews for years to come.
Futuremark may be 100% innocent with this async fiasco, but don't doubt that one of the reasons they're taking shit right now is because they have a criminal record and the Internet, like the North, never forgets.
It was not Futuremark's choice to evaluate CPU performance using a proprietary physics engine that didn't run on the CPU when GPUs from a single vendor were detected?
Whose choice was it then?

I was intrigued by this finding, so I did some research and have some follow up questions if you don't mind.
Ageia was acquired by nvidia in Feb 2008 according to this: hxxp://www.nvidia.com/object/io_1202895129984.html
While Vantage was released in April, with a still likely signed BDP contract (speculation, but it seems likely) which transferred to nvidia not two months before: hxxp://www.futuremark.com/pressreleases/49662
But regardless of that, PhysX cards in the wild were still the ones made by ageia (or whoever printed for them) and using ageia drivers.
So I was wondering when did actually nvidia assume control. And then found this: hxxp://archive.techarp.com/showarticlec777.html?artno=560&pgno=4 if you scroll down it says initial release happened on 20th of June for the dedicated physx drivers with nvidia logo on them.
Also note

Q : What cards work with the PhysX drivers?
A : The PhysX drivers work with GeForce 9800 GTX, GeForce GTX 280, and GeForce 260 GPUs. Drivers for all other GeForce 8 or GeForce 9-Series GPUs will be posted in July

Furthermore, the drivers were not incorporated into the whole WHQL driver package before 15th of October (maybe wrong, but I couldn't find anything earlier), hxxp://www.nvidia.com/object/winxp_178.24_whql.html
Earliest beta driver to support it aligns with the standalone physx driver: hxxp://www.nvidia.com/object/winvista_x86_177.39_beta.html

So considering all this info, Futuremark responded in one month (25th of june till 20th of july) of beta drivers, and way way before the release of WHQL drivers, likely with a signed BDP contract and the benchmark released two months after acquisition. Does this really pass as a criminal record? Isn't it much more likely they were caught of guard with the acquisition, and once they realized the situation they acted?

 

[Deleted member 13524]

Furthermore, the drivers were not incorporated into the whole WHQL driver package before 15th of October (maybe wrong, but I couldn't find anything earlier), hxxp://www.nvidia.com/object/winxp_178.24_whql.html
Earliest beta driver to support it aligns with the standalone physx driver: hxxp://www.nvidia.com/object/winvista_x86_177.39_beta.html

Those dates don't matter. The drivers that were sent to reviewers for the GTX 200 series had PhysX acceleration in June 2008, and that's what was used in reviews.

So considering all this info, Futuremark responded in one month (25th of june till 20th of july) of beta drivers, and way way before the release of WHQL drivers, likely with a signed BDP contract and the benchmark released two months after acquisition.

Futuremark didn't respond in one month. The acquisition was announced in February, it was then stated that PhysX would run on nVidia's GPUs so if anything Futuremark responded in 4 months.
If it was a contractual obligation from Ageia/nVidia like you seem to suggest, then they should have come clean with that subject. They didn't, which sounds like me being found next to a dead body and a gun in my hands and then get up and leave without saying anything.
And even if it was one month, anything past a couple of days for a statement in their website plus an e-mail being sent to reviewers to disable GPU acceleration on nvidia cards is downright unacceptable. Regardless, one month is (IMO) way too much to change a simple flag in order to avoid the acceptance of GPU acceleration and at the same time it's quite comfortably long enough to wait for all Tesla/R700 reviews to be published.




As for Time Spy itself, it's first showing had the nVidia logo splattered all over it:

I wonder what's the point of partnering with an IHV in order to develop a supposedly unbiased 3D graphics benchmark.
I get it when it's a game (free engineers, yay) but doing this in a synthetic benchmark just makes it irrelevant, IMO.

 

[3dilettante]

Well, that escalated quickly

I don't know much about interpreting GPUView, but is the objection that the benchmark has a modest amount of compute? Futuremark gave a 10-20% range in their guide?
Is the controversy over analysis that there's a preemption packet? The last time GPUView was discussed here, it indicated that if the rectangle was above the base layer that the command was sitting on the queue and not actively processing. The preemption block spends most of its lifetime queued up behind other commands before seemingly taking up a few pixels in the active stream. Perhaps it is a coarse last-chance barrier where the compute must finish before some important point in the next frame's graphics load?

 

[Esrever]

I wonder if their universe will implode when they find out, that the actual ALUs cannot run the compute queue in parallel to the 3D queue, but that at a wavefront level there still is timeslicing going on.

Just you wait for hyperthreaded ALUs!

 

[dogen]

I don't know much about interpreting GPUView, but is the objection that the benchmark has a modest amount of compute? Futuremark gave a 10-20% range in their guide?
Is the controversy over analysis that there's a preemption packet? The last time GPUView was discussed here, it indicated that if the rectangle was above the base layer that the command was sitting on the queue and not actively processing. The preemption block spends most of its lifetime queued up behind other commands before seemingly taking up a few pixels in the active stream. Perhaps it is a coarse last-chance barrier where the compute must finish before some important point in the next frame's graphics load?

The controversy doesn't seem to be about any one thing. Mr. Mahigan over at overclock.net claims that the benchmark is tailored for nvidia architectures. This is based on the fact that maxwell cards don't lose performance when turning async compute on, while other games like ashes do(this is debatable, the difference there is very small in that game). Now he goes into detail about fences and whatnot, saying that they would cause a performance loss whether async is disabled in the driver or not. I personally don't know enough to decide. Anyway, since there's no performance loss, they' must be doing it Nvidia Style. Whatever that means.

Now, the complaints include there not being enough async used and therefore it's not showing radeon cards' true potential. And that's what makes it biased. There's also the people throwing around GPUView screenshots coming up with whatever interpretations they think they mean.

Basically it's a big dumb mess, started by a few people making wild guesses based on incomplete information, and now the whole amd fanboy army has gone rabid.

 

[Florin]

Turns out I am:
Futuremark may be 100% innocent with this async fiasco, but don't doubt that one of the reasons they're taking shit right now is because they have a criminal record and the Internet, like the North, never forgets.

What fiasco? Criminal record? Huh?

 

[lanek]

As for Time Spy itself, it's first showing had the nVidia logo splattered all over it:

I wonder what's the point of partnering with an IHV in order to develop a supposedly unbiased 3D graphics benchmark.
I get it when it's a game (free engineers, yay) but doing this in a synthetic benchmark just makes it irrelevant, IMO.

I will need to add a little correction to this, This presentation was during the GOC event in Asia, Nvidia was the big sponsor of this event, and what you see on the upper zone is not really what is displayed by the benchmark or even on the display, but 2 logos separated over the screen where is displayed the benchmark video.

Thoses 2 logos was on the armature of this giant display and was lit all the time., but have nothing to do with the benchmarks..

here you have some screenshoot where it is more easy to seen it.
http://www.guru3d.com/news_story/futuremark_shows_first_footage_3dmark_directx_12.html

Honestly, i have not really follow what is happenning with Timespy, i was a bit busy this weekend..

 

[MDolenc]

What a time to be alive. Interesting times ahead..

Shameless plug but:
This were interesting times. This now is just bs.

 

[CarstenS]

Turns out I am:
https://www.techpowerup.com/66327/gpu-physx-doesnt-get-you-to-3dmark-vantage-hall-of-fame-anymore
When 3dmark Vantage released in April 2008, the "CPU Physics" test was using PhysX, which was GPU-accelerated if nvidia cards were detected. The total score was affected by a CPU test that was running on the GPU, giving nvidia cards a rather big advantage (mainly because PhysX ran like crap on CPUs at the time).

I'm pretty sure, that story is of similar thoroughly research background as most of the current debate. I will dig into it further, but only have time on hand over the weekend at the earliest. Thank you for understanding. If I am wrong, I have no problem with standing corrected. But the timeframe over Ageia's aquisition seems to imply that it's not Futuremarks malevolence that caused this.

From a short googling, this seems to support what I remember:
https://www.futuremark.com/support/troubleshooting
"Q:The physics score is invalid as PhysX GPU acceleration was used.
A:This message means your 3DMark Vantage benchmark run used PhysX GPU acceleration, running PhysX on the NVIDIA video card instead of on the CPU. You can fix this issue by selecting "Disable PPU" in the benchmark settings or by disabling GPU PhysX in the NVIDIA driver options, and then re-running the benchmark."
There WAS PPU acceleration for Ageia Cards in 3DMark Vantage, that seemingly got hijacked by Nvidia.

 

[Deleted member 13524]

If I am wrong, I have no problem with standing corrected. But the timeframe over Ageia's aquisition seems to imply that it's not Futuremarks malevolence that caused this.

What you disagreed with was not the accusation of malevolence, but the historical fact that 3dmark did allow for GPU-accelerated computation tasks contributing to what were otherwise CPU/PPU computation tasks in a CPU-centric test.


Each one can look at the timeline of events (also factual) and take their own judgment over the matter.
Though to be honest, I'm not the least bit interested in knowing everyone's single opinion on this almost 9 years later. The public opinion at the time was very clear, though.

 

[CarstenS]

What you disagreed with was not the accusation of malevolence, but the historical fact that 3dmark did allow for GPU-accelerated computation tasks contributing to what were otherwise CPU/PPU computation tasks in a CPU-centric test.


Each one can look at the timeline of events (also factual) and take their own judgment over the matter.
Though to be honest, I'm not the least bit interested in knowing everyone's single opinion on this almost 9 years later. The public opinion at the time was very clear, though.

Correct: 3DMark did allow.
What 3DMark did allow was using a PPU contributing to the final score.

You're not being interested in your opinion being contested just fits into the picture.

 

[Kaotik]

IIRC the whole Vantage/PhysX fiasco actually was caused by certain NVIDIA (beta?) drivers replacing the libraries used by Vantage to allow GPU acceleration, but need some googling to confirm

 

[Kaotik]

Well, that escalated quickly

Does that first post have any merit?

 

[Deleted member 13524]

Correct: 3DMark did allow.
What 3DMark did allow was using a PPU contributing to the final score.

Wrong again, it was PPUs and nvidia GPUs:

3DMark Vantage is obviously fresh from the shelves. With all the mayhem of PhysX results ending up in the scores and what not, we now officially ditch the "overall P" score.

And BTW while Futuremark filtered out the GPU-PhysX in the Hall-of-Fame, it seems it would still appear turned on by default in the presets, for years to come:

You're not being interested in your opinion being contested just fits into the picture.

Oh you may contest it just fine and you might be right in some of your assumptions (even though you have been wrong in some facts).
It just doesn't matter for the topic at hand and it's just further thread derailment. Maybe you can create a topic for that discussion elsewhere?



The only reason why I brought the issue up is because Futuremark suffered a huge public backlash back then and everyone became suspicious of them. Either you want to say/prove that the enthusiast communities were wrong for being angry is irrelevant now, because won't change the fact that they were angry and distrustful of Futuremark. And this episode weighs in on the general trust of Futuremark even now.

Yes, I know many intellectual(loid)s on B3D want to blame 100% of the current accusations against Futuremark on blind AMD fanboyism from all those inferior ignorant plebs. But although those plebs may be wrong (they many times are), the world isn't black and white.

That said, my point about how this suspicion comes partially from Futuremark's past has been made and still stands. I won't touch the PhysX issue any further.

 

[Razor1]

The controversy doesn't seem to be about any one thing. Mr. Mahigan over at overclock.net claims that the benchmark is tailored for nvidia architectures. This is based on the fact that maxwell cards don't lose performance when turning async compute on, while other games like ashes do(this is debatable, the difference there is very small in that game). Now he goes into detail about fences and whatnot, saying that they would cause a performance loss whether async is disabled in the driver or not. I personally don't know enough to decide. Anyway, since there's no performance loss, they' must be doing it Nvidia Style. Whatever that means.

Now, the complaints include there not being enough async used and therefore it's not showing radeon cards' true potential. And that's what makes it biased. There's also the people throwing around GPUView screenshots coming up with whatever interpretations they think they mean.

Basically it's a big dumb mess, started by a few people making wild guesses based on incomplete information, and now the whole amd fanboy army has gone rabid.

First off Mahigan doesn't know what parallel and concurrent execution, looks like he typed it right out of wikipedia . Without knowing this I don't know how the f they talk about fences and barriers lol. Then you have another group of guys that can't even read Gpuview trying to decipher it lol, that is just funny as hell. As you stated async is disabled on a driver level for maxwell so yeah there won't be a change in performance for their cards.....

PS OC.net guys, since I won't post over there, like ever lol, if you are reading this, there is no preemption going on, Figure out How to Read GPUview before you start making assumptions, cause once you do that you don't need to make assumptions lol. ALSO DO NOT PUT MARKETING MATERIAL WITH your findings from GPUview, cause I can tell you right now they don't line up at all and you guys are creating a shit storm by doing so, cause you don't understand what you are talking about, how the hell do you think others on your forum or people with even less knowledge will take it? And this has already happened IDIOCY running rampant is not a good thing for anyone! Yeah it it starts from the people that started this crap.

 

[Andrew Lauritzen]

Just you wait for hyperthreaded ALUs!

GPU execution units are already heavily "hyper-threaded" with 8+ hardware threads/unit being very normal

 

[troyan]

Futuremark released their statement about Async Compute in Time Spy: http://www.futuremark.com/pressreleases/a-closer-look-at-asynchronous-compute-in-3dmark-time-spy

 

[surreal]

Those dates don't matter. The drivers that were sent to reviewers for the GTX 200 series had PhysX acceleration in June 2008, and that's what was used in reviews.



Futuremark didn't respond in one month. The acquisition was announced in February, it was then stated that PhysX would run on nVidia's GPUs so if anything Futuremark responded in 4 months.
If it was a contractual obligation from Ageia/nVidia like you seem to suggest, then they should have come clean with that subject. They didn't, which sounds like me being found next to a dead body and a gun in my hands and then get up and leave without saying anything.
And even if it was one month, anything past a couple of days for a statement in their website plus an e-mail being sent to reviewers to disable GPU acceleration on nvidia cards is downright unacceptable. Regardless, one month is (IMO) way too much to change a simple flag in order to avoid the acceptance of GPU acceleration and at the same time it's quite comfortably long enough to wait for all Tesla/R700 reviews to be published.




As for Time Spy itself, it's first showing had the nVidia logo splattered all over it:

I wonder what's the point of partnering with an IHV in order to develop a supposedly unbiased 3D graphics benchmark.
I get it when it's a game (free engineers, yay) but doing this in a synthetic benchmark just makes it irrelevant, IMO.

Now I noticed, indeed they didn't respond in one month, it was in two weeks as can be seen here:
https://forum.beyond3d.com/threads/...-results-now-removed-from-hall-of-fame.43342/
Considering it's a Nordic company, two weeks right in their summer vacation. I'd say this is not bad at all if you know what I mean
Then I really read into the content of the post (you can retrieve the link on waybackmachine), and it would seem like NVidia did something dirty. I can't imagine the guys at Futuremark were approving of that move.
I also found Comment #18 on your first link is interesting, there's a quote (link is dead):

NVIDIA's GPU PhysX was launched in the middle of summer and most of FM's staff are on holidays (Most people are here in Finland). Tero said timing couldn't have been worse but they have updated filtering system coming up which will handle the runs which are not following Futuremark's guidelines.

When I asked more specific about NVIDIA's GPU PhysX, he confirmed that it won't be allowed. (Since Vantage's CPU test 2 is designed to measure CPU physics calculations and it is clearly stated in the rules that GPU or driver can't affect the result significantly).

So from now on if you want to run offical 3DMark result or Hall of Fame result with NVIDIA gfx you have to use Futuremark approved WHQL driver and do not install NVIDIA PhysX System Software.

It is quoted two weeks after the fact, but still provides some insight into the situation.

Furthermore, I found the commenters on your link quite upset about the removal of PhysX from valid scores rather than of its inclusion in the first place. Doesn't really coincide with the backlash. I mean it does, but only if the backlash was directed at Futuremark for invalidating PhysX scores, which makes this discussion moot anyway.

Wrong again, it was PPUs and nvidia GPUs:

Apparently not so much, at least not as Futuremark intended and they addressed it quite quickly.