Shader Compilation on PC: About to become a bigger bottleneck?

[Remij]

Agreed. We already have to wait an hour+ for download and install on a decent internet connection. What does another 10-15m matter in the grand scheme of things.

Exactly. That's why I brought up that we have to wait to download a game, and have to wait again to download it if we delete it to make room on our storage drives. It's a necessary evil, and we wait that time with no issues what so ever.

Having a 10min process happen once the very first time you launch a game, and most likely never again..or very rarely, is absolutely nothing in the grand scheme of things. Now if your game is performing like junk and hitching all the time.. that's a far more detrimental to the experience.

People just have to think of it like an extended install process. One phase where you download the game, and the other where you need to run it to finish "installing" it.

 

[Flappy Pannus]

Agreed. We already have to wait an hour+ for download and install on a decent internet connection. What does another 10-15m matter in the grand scheme of things.

This comparison would be applicable if you had to download your games anew after every driver or US update.

One of the biggest complaints about the Epic launcher btw is that it can't simply recognize currently existing installed games from a previous OS installation, if for example you took away Steam's ability to just recognize games that are already installed and do a 5-minute verification pass but required you to re-download them every time, even with the relative infrequency of OS reinstalls/updates you would never hear the end of the screaming on the forums.

 

[Flappy Pannus]

. Now if your game is performing like junk and hitching all the time.. that's a far more detrimental to the experience.

If simply doing shader pre-compilation as a whole before the game launched was a panacea for game hitching on PC (as you seem to believe), I would probably be far more acceptable of it.

But...it isn't? There are many reasons that a game can be stuttering on PC. H:ZD has a length shader compilation process - it still stutters like crazy on many configs atm. Detroit:Beyond Human pre-compiles its shaders in a long process as well, it can still stutter. I can get stuttering in Dishonored 2 due to texture paging that are not related to shader compilation, which it does on load frequently (even outside of driver updates - and again, as we're seeing with HZD, re-engaging shader compilation automatically on driver updates is the least-frequent implementation possible). Arkham Knight's stuttering is not due to shader compilation either.

We agree that stuttering in-game is far more detrimental to the experience than a lengthy pre-compiling stage, obviously as a % of gametime, even if I play the game interspersed between driver updates and have to see that compiling screen several times before I complete it, it comes out to a relatively small impact overall relative to the time I wait in game loading other elements. I just see little evidence that introducing the pre-compiling step is some sort of stuttering cure-all.

More to the point of my first post though, is that we're soon going to have consoles that don't have this problem, but also will not have the load time issues that they do now inside of the game. As said, while the lengthy pre-compile process is annoying, atm at least compared to consoles, the overall experience of waiting for game assets to load on PC is far superior in terms of returning to the action after a death, fast traveling, etc - game like Doom 2016 for example is far more enjoyable on hard difficulties on my PC over my PS4 in part because when I die I'm back in the action in seconds.

But very soon, that will no longer be a distinction, in fact it will likely go the other way, especially with consoles implementing state resume. It's one thing to no longer have a large advantage for a more expensive platform, it's another to actually have a detriment. I wonder if a gamer used to bouncing between games on their PS5/Xbox Series X in seconds loads up a game in PC, and after waiting for that game to actually load (as there is no save state functionality), then has to stare at a 'recompiling shaders' screen for 15-20 mins. I can imagine them thinking 'wtf is this shit?'. I think that's a problem.

 

[MfA]

With 8+ cores and proper storage QoS why should it cause stuttering to begin with? Assuming it pre-compiles all the shaders necessary for the current level (or local zones for open world) and then does the rest in the background.

It seems more a failure of the code and/or the OS, than background compilation in principle.

 

[Lurkmass]

Somewhat unrelated but there is a way to improve the situation which is to expose some API extension like having more dynamic rendering states available in the drivers. This can help reduce the number of PSOs needed because more of the graphics state can be set outside of the pipeline and as a result it cuts the length of the shader compilation time as well but the added irony is that this pretty much goes against the stateless design of the new gfx APIs which was what led to the downfall of the old gfx APIs ...

Nevertheless, I trust that the HW vendors know what they're doing with that extension since they can at least guarantee that changing those rendering states won't cause shader recompilations behind our backs ...

 

[Ethatron]

To me it's a failure of the market principles. As this mechanism becomes more complex, it should be distributed to a player that focuses on only solving these aspects, and provide a stable, useful middle-ware. Besides the big engine players adopting other middle-wares, there is a broken development culture going on, preventing these adoptions. "It cost", "We can do it", "We can do it better", "We can't deal with the bugs", "Asking for new features and corrections takes aaages", etc. blabla. Then some IHVs think they can gain a competitive advantage to give out utility libs/code, but it only works on their hardware, or in their eco-system, is even more unconstructive noise in an already exhausting dev cycle.

In terms of collaborative development of rock solid foundations/solutions for complex multi-platform and multi-hardware management issues and programming patterns, game development is not coping well in the PC space. Some console vendors are very good, they give you huge amounts of utility/code, which they improve and which is the (potential) baseline for almost every game on their platform. Others not so, they make it their philosophy to give freedom to everyone to implement it in their own needed optimal way, feeding into the broken development culture dead-loop above, and apparently consciously taking into account that QA for their platforms is all over the place.

Many challenges are larger than one game developer, and sadly, over short or long term, the big engine players will be the only ones able to provide proven Quality of Service for their lower level sub-systems behaviour, and all the others will have to "cope". It's sad, as there is very little counter-movement towards constructive collaboration, that I can see. One big good example is DXC and the SPIR-V tools, it blooms and it's great (from a culture perspective).

 

[Alessio1989]

All the more reason for there to be a game shader by driver repository where then consumers merely need the game to just download the appropriate package.

What could be going wrong? oh wait anything XD

The real question is: what kinda shader sources provides those games, IL/bytecode or just source/assembly?

 

[Per Lindstrom]

Interesting topic.
One way to implement it, could be a right-click on game-icon, and then select compile shaders in background.

 

[Per Lindstrom]

With 8+ cores and proper storage QoS why should it cause stuttering to begin with? Assuming it pre-compiles all the shaders necessary for the current level (or local zones for open world) and then does the rest in the background.

It seems more a failure of the code and/or the OS, than background compilation in principle.

Lack of memory?
DX12 need more memory then DX11, i have been told.
Maybe this is one reason?

 

[MfA]

If a shader compiler needs lots of memory something is very wrong with the implementation too.

 

[Per Lindstrom]

If a shader compiler needs lots of memory something is very wrong with the implementation too.

It may be so, that's the case for Horizon Zero Dawn.
12 GB is extreme, so yes, something is very wrong.

https://www.reddit.com/r/pcgaming/comments/i5towt

 

[MfA]

No excuse really, this isn't compiling desktop spaghetti code. Someone needs to fix their shit.

I don't see how it would be impossible to use resource limits to keep the background compilation at a level where it should not impact the consistent performance of the game. It might consistently lower it a bit, but it shouldn't jump up and down.

 

[Flappy Pannus]

Interesting topic.
One way to implement it, could be a right-click on game-icon, and then select compile shaders in background.

That's actually an interesting idea, perhaps games would report to Windows when they detect shader compilation is necessary and you have an option in Windows settings you can enable to 'compile game shaders during idle', basically a low-priority process that would fire up when necessary. You'd definitely have to be careful to make it get the hell out the way when you actually need those CPU cycles though, it's one thing to get shader stuttering in the game it's actually compiling for, it's another to induce stuttering for compiling shaders for a game you're not even playing.

 

[Remij]

That's actually an interesting idea, perhaps games would report to Windows when they detect shader compilation is necessary and you have an option in Windows settings you can enable to 'compile game shaders during idle', basically a low-priority process that would fire up when necessary. You'd definitely have to be careful to make it get the hell out the way when you actually need those CPU cycles though, it's one thing to get shader stuttering in the game it's actually compiling for, it's another to induce stuttering for compiling shaders for a game you're not even playing.

That's what I mean. Just give me the option to do it, and I'll absolutely take advantage of that.

What's 10 or 15min of an "install process" added on to hours long download times for some people?

It's when you click "play" and you expect to just be able to get into the game and THEN have to go through another process is when it starts to annoy people.

Making it a part of the Steam Download/Install process would likely solve most of the arguments with it.

 

[Flappy Pannus]

Making it a part of the Steam Download/Install process would likely solve most of the arguments with it.

Doing that would help a bit, but the big issue with shader compilation is it needs to be done with every driver update, and in some cases, game patches. If it just needed to be run once during an install I likely wouldn't have ever thought of even creating this thread.

 

[Remij]

Doing that would help a bit, but the big issue with shader compilation is it needs to be done with every driver update, and in some cases, game patches. If it just needed to be run once during an install I likely wouldn't have ever thought of even creating this thread.

Well once again, with game patches the process could be done through the Steam process.. and I still honestly don't see people changing drivers often through a single playthrough of a game. Once a month you have to deal with a 5-10min "optimization"... not a big deal at all, and a worthy sacrifice for smoother gameplay. Again, that's just me.

So if you essentially hide 2/3rds of the times when this process would need to happen, then you're still better off than you were before.

 

[Per Lindstrom]

One strategy can be to compiling a slow version of the shaders in short time, and then pop up a question window.

Your Shaders, for this game, are not optimized, this takes rather long time.
Optimized Shaders increase performance.

Do you want to?

* Play now, you will be asked if you want to optimize Shaders, after finished playing.

* Play now, and optimize Shaders while you play, this may decrease game performance, during optimization.

* Optimize Shaders before you play, you may use your computer as usual, during optimization.

    (OK)   (Cancel)

 

[Davros]

question : with hundreds of games installed why do i have no memory of these incredibly long delays caused by shader compilation ?

 

[BRiT]

question : with hundreds of games installed why do i have no memory of these incredibly long delays caused by shader compilation ?

You've been buried in your TB of games from the 90s? Not running DX 12 enabled software and hardware?

 

[DegustatoR]

question : with hundreds of games installed why do i have no memory of these incredibly long delays caused by shader compilation ?

Because most games manage to do it properly.