No DX12 Software is Suitable for Benchmarking *spawn*

[CarstenS]

Why exactly are we ignoring Doom, which performs better with the low level API? Or are we keeping these comparisons limited to only tacked on DX12 ports that aren't expected to have ideal performance?

I have no idea why you're so against the low level APIs. They've already won and all the engines are moving in that direction. Just because current engines are largely hamstrung by DX11 requirements doesn't mean they are flawed.

I'm not against Low Level APIs, far from it: using a quadcore cpu in my gaming rig, I welcome all the CPU performance I can get. And I agree, the new APIs are the future and are destined to be widely used. BUT, the problem is with the applications, namely many DX12 games that fall short of what's expected. Moreover, the problem is, that quite a lot of integrated benchmarks seem to imply the opposite, painting a picture that's coming true in the future - in an ideal world.

Doom - you're right here - is a very good example of LL-API integration. It's not directly connected to the thread title though, maybe a mod can fix this?

They definitely seem to have some pipeline issues. Too many fences or bad dependencies would seem logical.

Would love to see some kind of analysis about that. Strangely, they already cut DX12 from the launch features - why not wait now until it works as it should (I'm guessing here, but I don't see how the developers could wish for this).

 

[sebbbi]

The reason why we don't see bigger performance boost in DX11->DX12 ports is that most engine code bases are designed around DX11 (DX9 roots). If you simply implement DX12 behind an existing graphics API wrapper, you will basically end up emulating the DX11 driver. That's not going to bring magical performance boosts. You need to completely rewrite your whole resource management pipeline, including data streaming, loading, memory management, dynamic resources, etc. You need to implement logic to group draws in bundles (and cache them) instead of submitting lots of tiny new draw calls every frame. You need to use ExecuteIndirect when drawing lots of bulk objects with similar state (such as shadow maps). And you need to use bindless resources to reduce draw call counts (bindless makes all resources are available without binding -> resource changes don't need draw call boundaries). None of the current DX12 PC games use bindless resources. The reason is that most engines are so tightly programmed around the old DX9/10/11 binding model. Changing that would require a major rewrite to the engine code base.

The biggest problem is: Major rewrite to support DX12 perfectly (bindless and all) would make the code base incompatible with the old DX9/DX11 binding and resource management model. Dropping DX11 support is not an option for most developers, as Windows 7 is still so widely used. Vulkan (supporting Windows 7) might be the black horse, allowing developers to move faster to these radically new rendering engine designs. As soon as we get fully featured robust HLSL -> SPIR-V compiler (it is bound to happen), there is nothing preventing studios in dropping the legacy DX11 support. That's when we see big changes. IMHO Microsoft made a big mistake when they decided to limit DX12 to Windows 10. Many developers would have already dropped DX11 (in favor of DX12) if DX12 supported Windows 7.

 

[DavidGraham]

The reason why we don't see bigger performance boost in DX11->DX12 ports is that most engine code bases are designed around DX11 (DX9 roots).

Can we count native DX12 titles in that list as well? the likes of Quantum Break, Forza and Gears? XO exclusive titles developed with a low level code ported exclusively to PC DX12?

Dropping DX11 support is not an option for most developers, as Windows 7 is still so widely used

You need to also cater for hardware demographics, for example going purely DX12 will mean cautiously assigning bandwidth around the limitations of PCI-E gen 2, as it is still common to this day. Another example is different DX12 hardware Tiers, It's a mess out there, an issue Futuremark struggled a bit with in making their benchmark.

You need to implement logic to group draws in bundles (and cache them) instead of submitting lots of tiny new draw calls every frame

Some developers state DX12 bundles are not really faster on the GPU side, as the gains from optimizing them is mostly CPU side. Some has gone as far as specifying their use on AMD hardware only if the game is mainly CPU bound. What do you think about that?

As soon as we get fully featured robust HLSL

The use of root signature tables (where AMD and NV diverge massively optimization-wise) can be integrated within HLSL, Will that integration mitigate at least some of that divergence?

 

[sebbbi]

Can we count native DX12 titles in that list as well? the likes of Quantum Break, Forza and Gears? XO exclusive titles developed with a low level code ported exclusively to PC DX12?

Gears is using Unreal Engine. Unreal Engine 4 is designed around traditional (DX11) binding model. Quantum Break developers (Remedy) had a full presentation talking about their DX12 porting process. They were not DX12 from the beginning. I don't know details about Forza, but the previous Forza game was released before DX12, and I am pretty sure they didn't rewrite their whole graphics engine between the last two games.

Ashes of Singularity is one of the only true DX12 titles. Nitrous engine was designed for DX12 from the ground up. Their Star Swarm stress test already showed huge performance gains on DX12. And even they have DX11 legacy code path (limiting their options). We have to wait for engines designed around ExecuteIndirect and bindless resources. Console exclusives of course can already use these features, but porting to PC is going to be tricky, unless you can afford to be Windows 10 only.

 

[function]

Can we count native DX12 titles in that list as well? the likes of Quantum Break, Forza and Gears? XO exclusive titles developed with a low level code ported exclusively to PC DX12?

I wouldn't really assume that console games can afford to throw away their entire codebase either. Games that run on middleware couldn't even if they wanted to. Gears is on Unreal, so you can scratch that off the list immediately.

You need to also cater for hardware demographics, for example going purely DX12 will mean cautiously assigning bandwidth around the limitations of PCI-E gen 2, as it is still common to this day. Another example is different DX12 hardware Tiers, It's a mess out there, an issue Futuremark struggled a bit with in making their benchmark.

PCI-E bandwidth seems to be far less of an issue than latency and driver overhead for transferring data across. BW is already massively higher than what you'd need to support texture and mesh data being moved in on an "upcoming frames" basis. In that regard, DX12 should help massively ... once you've built a game around efficient streaming of data in. Which you won't do, while you're supporting DX11 ...

Some developers state DX12 bundles are not really faster on the GPU side, as the gains from optimizing them is mostly CPU side. Some has gone as far as specifying their use on AMD hardware only if the game is mainly CPU bound. What do you think about that?

So use less CPU and use the power to turbo the GPU higher. Hooray, you're faster on the GPU side now!

 

[Anarchist4000]

I'm not against Low Level APIs, far from it

I was referring to someone else.

Would love to see some kind of analysis about that. Strangely, they already cut DX12 from the launch features - why not wait now until it works as it should (I'm guessing here, but I don't see how the developers could wish for this).

One of the benchmarks was showing identical performance between 1080 and 1440 as I recall. Even waiting for the entire frame to finish prior to starting the next I'd expect at least minor differences there. For whatever reason it seems they aren't dispatching frames quickly enough and always waiting on something to complete. I'm not sure how much analysis will help there outside of having the actual game code. Maybe someone could make sense of the fences, but it seems more likely to be their threading model.

And Doom is no exception to the rule, it's game with a tacked on lower level API, yet it succeeded in achieving some of the promised features (not all though, and still had it's own problems), and thus has traveled further than most of the poor DX12 titles. So the argument that games with tacked on APIs are inherently bad and not representative of the true form is quite baseless. There you have an example that is just that, achieving better results.

It wasn't really tacked on though. OGL was working on AZDO (low overhead) models back in 2014. That tracks pretty well with when they would have been developing their engine. It's also a fair approximation of Vulkan/DX12. They arguably took it a step further with the 50+ render targets and asynchronous updating of resources. And by asynchronous I mean only updating the invalidated portions with some resources persisting for many frames.

 

[Deleted member 2197]

12.09.2016
http://www.pcgameshardware.de/Deus-.../Specials/DirectX-12-Benchmarks-Test-1207260/

 

[CSI PC]

12.09.2016
http://www.pcgameshardware.de/Deus-.../Specials/DirectX-12-Benchmarks-Test-1207260/

At least that now confirms what Computerbase.de noticed with the internal benchmark skewing results.
Sadly though it is like memes with so many just use the results of the internal benchmark to base their decisions/conclusions and spreading it onwards.
I think it is time for publications to start to be pretty vocal about said internal benchmarks, and that complete transparency/disclosure should be provided by developers when they include them.

Cheers

 

[I.S.T.]

Can someone summarize that for the not german speaking folk here?

 

[Razor1]

what I gather from google translate, pretty much as compute needs increase (higher quality shaders) Aynsc performance increases on both IHV's and AMD gets a higher boost from this, while playing the game too, but there is a huge difference from the internal benchmark vs the game play.

So it seems AMD is optimizing specifically for the benchmarks and not the game. Same story different day

DX12 performance is significantly worse across all boards than DX11 while playing

 

[iroboto]

Unsure as to how long it takes to re-write the renderer to be ground up DX12. But considering the number of tools they may have to rebuild. Ayyyyyy.

Graphics performance is one factor, but time is money. Content creation is super time consuming. I do see this as a hard transition for many companies and they are doing the best that they can to cope.





Sent from my iPhone using Tapatalk

 

[lanek]

what I gather from google translate, pretty much as compute needs increase (higher quality shaders) Aynsc performance increases on both IHV's and AMD gets a higher boost from this, while playing the game too, but there is a huge difference from the internal benchmark vs the game play.

So it seems AMD is optimizing specifically for the benchmarks and not the game. Same story different day

DX12 performance is significantly worse across all boards than DX11 while playing

Is it AMD or Nixxes who have for the preview fix more problem on the benchmarks and not in the game, because it seems to be the same story for Nvidia. ( untill AMD optimize for nvidia )

 

[Razor1]

nV's results from both DX11 and DX12, game play are higher than the benchmarks results

 

[Deleted member 13524]

All this recent demonization of internal benchmarks (made mostly by the BFFs in this subforum) is ridiculous. And of course this had to appear in a time when the new API shows one vendor consistently getting considerable performance advantages to their competitors within the same price range.


It's their new thing. "B-but AMD is optimizing only for the internal benchmarks".
Yeah, everyone is picturing AMD >6 years ago when laying down the plans for GCN having secret meetings with Square-Enix to optimize their architecture for a runtime demo of an unannounced game yet to start production while running under a Beta implementation of an unannounced API. Not ridiculous at all.
Even suggesting they're doing heavy driver optimizations for the internal benchmarks is ridiculous because DX12 weighs a lot less on the driver and a lot more on the hardware and the game's own code. Either the hardware performs adequately to the game/engine or it does not.
Though everything is permitted when trying to follow the narrative, I guess.


Scripted internal benchmarks are the only practical way to guarantee that all system setups are going through the exact same loads.
Does computerbase pcgameshardware use a scripted keyboard+mouse input for their "own" benchmark and do they publish video recordings of said benchmarks showing the exact same playthroughs with the exact same number of polygons and particles on screen?
If not, then computerbase's pcgameshardware's "non-internal" benchmark results mean utter crap because no one will ever know if they're playing the whole run facing the floor or a wall on AMD cards and facing the scenery on nvidia cards, or vice versa.
Even if they did use scripted KB+M inputs, in-game NPC AIs rely on random variables so the end process would still never be the same.


It's not by pure chance that all scientific papers in respected publications must always provide the readers with methods to accurately reproduce their findings, either through descriptions in the paper itself or through references with said descriptions. Otherwise, it's worth shit.


Where was all this condemnation of internal benchmarks when we were looking at previous games until not long ago, like GTA V, Arkham Knight, every single Far Cry game to date, etc.?
And now with DX12 the internal benchmarks are suddenly a dirty word?


But nevermind this little bit of sanity. By all means, carry on with these conspiracy theories.

EDIT: Not computerbase. It was pcgameshardware. Wrong german site.

 

[Razor1]

Unsure as to how long it takes to re-write the renderer to be ground up DX12. But considering the number of tools they may have to rebuild. Ayyyyyy.

Graphics performance is one factor, but time is money. Content creation is super time consuming. I do see this as a hard transition for many companies and they are doing the best that they can to cope.





Sent from my iPhone using Tapatalk

well its a hard transition but its one they will do, as their next engine version will drop DX11.

 

[Razor1]

All this recent demonization of internal benchmarks (made mostly by the BFFs in this subforum) is ridiculous. And of course this had to appear in a time when the new API shows one vendor consistently getting considerable performance advantages to their competitors within the same price range.


It's their new thing. "B-but AMD is optimizing only for the internal benchmarks".
Yeah, everyone is picturing AMD >6 years ago when laying down the plans for GCN having secret meetings with Square-Enix to optimize their architecture for a runtime demo of an unannounced game yet to start production while running under a Beta implementation of an unannounced API. Not ridiculous at all.
Even suggesting they're doing heavy driver optimizations for the internal benchmarks is ridiculous because DX12 weighs a lot less on the driver and a lot more on the hardware and the game's own code.
Though everything is permitted when trying to follow the narrative, I guess.


Scripted internal benchmarks are the only practical way to guarantee that all system setups are going through the exact same loads.
Does computerbase use a scripted keyboard+mouse input for their "own" benchmark and do they publish video recordings of said benchmarks showing the exact same playthroughs with the exact same number of polygons and particles on screen?
If not, then computerbase's "non-internal" benchmark results mean utter crap because no one will ever know if they're playing the whole run facing the floor or a wall on AMD cards and facing the scenery on nvidia cards, or vice versa.
Even if they did use scripted KB+M inputs, in-game NPC AIs rely on random variables so the end process would still never be the same.


It's not by pure chance that all scientific papers in respected publications must always provide the readers with methods to accurately reproduce their findings, either through descriptions in the paper itself or through references with said descriptions. Otherwise, it's worth shit.


Where was all this condemnation of internal benchmarks when we were looking at previous games until not long ago, like GTA V, Arkham Knight, every single Far Cry game to date, etc.?
And now with DX12 the internal benchmarks are suddenly a dirty word?


But nevermind this little bit of sanity. By all means, carry on with these conspiracy theories.

Dude I suggest you post else where if you are going to come in here with that kind of attitude.

We have seen both IHV's manipulate internal benchmarks many many times over two decades of this. This is not new!

Scripted benchmarks are the easiest thing to manipulate, they can profile it and they will know exactly what it coming and plan for it in driver.

 

[Silent_Buddha]

Unsure as to how long it takes to re-write the renderer to be ground up DX12. But considering the number of tools they may have to rebuild. Ayyyyyy.

Graphics performance is one factor, but time is money. Content creation is super time consuming. I do see this as a hard transition for many companies and they are doing the best that they can to cope.

Probably not full on ground up Dx12, but I think we'll start seeing a lot more of things like this where developers look to iterate by replacing Dx11 centric rendering with Dx12 centric rendering piece by piece while the engine overall remains based on Dx11.

https://forum.beyond3d.com/threads/...a-useful-reference.46956/page-13#post-1932825

But as I remarked in that thread, it was only talking about one area of the engine that was demonstrated some new techniques that showed impressive gains with Dx12. It's unkown if the engine is still based on Dx11 with tacked on Dx12 or if it's Dx12 with a Dx11 path.

It's basically engine research meant for as yet unannounced titles.

I don't see most engines going full on Dx12 with a Dx11 path for at least a few years. I think we'll see more of an approach like this, with each iteration of an engine getting increased support for Dx12 centric rendering while still being Dx11 at its base.

That will still be a rather large change from the majority of current titles that have very rudimentary, suboptimal and tacked on Dx12 support.

Regards,
SB

 

[Deleted member 13524]

Scripted benchmarks are the easiest thing to manipulate

Oh ok. Then it's the cash-less AMD who is sending cash to all developers who are implementing DX12 paths to manipulate - not the whole game/engine - but just the internal benchmark.
Because intentionally putting the IHV with 70% of the discrete GPU marketshare in a (false) bad light is totally worth it in exchange for these millions of cash that AMD is giving to them.
It's not like the dev and the publisher are interested in optimizing their game for 70% of their customers or anything.

 

[Razor1]

Oh ok. Then it's the cash-less AMD who is sending cash to all developers who are implementing DX12 paths to manipulate - not the whole game/engine - but just the internal benchmark.
Because intentionally putting the IHV with 70% of the discrete GPU marketshare in a bad light is totally worth it in exchange for these millions of cash that AMD is giving to them.
It's not like the dev and the publisher are interested in optimizing their game for 70% of their customers or anything.

What they don't need to manipulate the dx12 paths on the dev's end. Not even the whole engine.

All they have to do is look at one shader or two, and manipulate those.....

Have you seen HL2's water shader, on ATI cards its performed great, on nV cards at the time not so great, and it was just one instruction change that caused that! (it was found by a guy that didn't even have access to the source code)

nV has done the same thing too with their sponsored titles, if you don't believe me just search with google, cause you will find em.

I suggest you do you homework before you start talking about these things. Or at least don't come on here with that kind of attitude. This is the second time in the past week you are instigating something.

I could post an ostrich with its head in the sand but I won't.

 

[DavidGraham]

when the new API shows vendors consistently getting considerable performance degradation

Fixed that for you.

Even suggesting they're doing heavy driver optimizations for the internal benchmarks is ridiculous because DX12 weighs a lot less on the driver and a lot more on the hardware and the game's own code. Either the hardware performs adequately to the game/engine or it does not.

Apparently not, it still depends very heavily on the driver. AMD themselves released a DX12 specific driver for Deus Ex to coincide with the DX12 patch. Also apparently all those great posts explaining why there is really almost no true DX12 implementations as of yet have gone completely unnoticed by you.

Scripted internal benchmarks are the only practical way to guarantee that all system setups are going through the exact same loads.

While true in theory, they shouldn't give opposite results to normal gameplay scenarios, as presented in the Deus Ex example, where AMD DX12 causes stutters, jitter and latency spikes measured by 3 independent publications, in addition to fps loss. Which makes the experience worse than NV.

Some built-in benchmarks don't even use 100% identical scenarios, like Ashes for example. You will discard those as well?

Where was all this condemnation of internal benchmarks when we were looking at previous games until not long ago, like GTA V, Arkham Knight, every single Far Cry game to date, etc.?

All those examples show consistency between gameplay and built-in tests. If one vendor is faster in the benchmark it will remain faster in gameplay and vise versa.

If not, then computerbase's "non-internal" benchmark results mean utter crap because no one will ever know if they're playing the whole run facing the floor or a wall on AMD cards and facing the scenery on nvidia cards, or vice versa.

Oh, and so is Techreport and pcgameshardwre as well? Yeah the whole lot of review sites are now utter crap because they don't fit your fallen from the sky AMD narrative, right?