Shader Compilation on PC: About to become a bigger bottleneck?

[Remij]

@Dictator I saw your post on Bsky lol. I hope you guys cook this issue specifically and really call attention to SquareEnix to fix it in your video. This is a fixable issue on their side.. but we all know SquareEnix don't exactly update their games often.. and when they do, it's usually rubbish fixes and nothing substantial.

I was really hoping they'd update the original game to precompile shaders.. but alas they haven't.

Right now their eyes are on this game, and the longer we wait, the less likely it gets that they'll fix it.

We REALLY need a Fossilize equivalent for DX12 at this point. If MS doesn't do something like this which would allow Steam and other platforms to host PSO repositories like Vulkan does.. then I'm just biding time until SteamOS works on Desktop. As soon as it does, I'm out. Fuck this.

 

[Remij]

Big post on Unreal Engine's official site about Game Engines and Shader Compilation stuttering and their solution to the problem!

Goes into some detail on why it's a problem, what PSOs are, how the engine handles them, their new precaching system, and future work.

https://www.unrealengine.com/en-US/tech-blog/game-engines-and-shader-stuttering-unreal-engines-solution-to-the-problem

Also, they say right at the bottom that they will also be discussing this topic on the Inside Unreal stream this Thursday (Feb 6th)!

 

[Flappy Pannus]

Alex is on Bsky now, has a thread on a recent Sony State of Play and the preponderance of UE5 titles.

While UE's issues extend beyond just gathering PSO's, I'm sticking this in this thread as the point about it being improved/solved in the latest version of UE5 means that we will still see many games with missed PSO's in the coming years as it's just not tenable to switch to the latest version of an engine at a certain point in development.

This is why I really hope we see sort of Fossilize system being implemented by an IHV or MS, something where gamers can basically crowdshare discovered PSO's for local processing. Even for games that have a thorough compilation process, it's still beneficial if it can be done outside of gameplay so you're never stuck waiting for 5-10 minutes at a shader compilation screen.

 

[homerdog]

I just fired up Enshrouded again and it does shader comp well. It starts when you enter the game and tells you the progress and how many threads it's running on. You can play the game while it's running but it states that it will go faster if you let it finish before playing. After letting the process finish, I started playing and didn't notice any stutters. Took around 8-12 minutes (I didn't time it) on a 13600K and claimed to be using 8 threads. Dunno how/why it decided on 8 but Windows correctly decided to load up all 6 P cores and bounce the remaining threads between the E cores.

 

[Remij]

Alex is on Bsky now, has a thread on a recent Sony State of Play and the preponderance of UE5 titles.

While UE's issues extend beyond just gathering PSO's, I'm sticking this in this thread as the point about it being improved/solved in the latest version of UE5 means that we will still see many games with missed PSO's in the coming years as it's just not tenable to switch to the latest version of an engine at a certain point in development.

This is why I really hope we see sort of Fossilize system being implemented by an IHV or MS, something where gamers can basically crowdshare discovered PSO's for local processing. Even for games that have a thorough compilation process, it's still beneficial if it can be done outside of gameplay so you're never stuck waiting for 5-10 minutes at a shader compilation screen.

Exactly. We need a multi-pronged approach to essentially catch most edge cases to be able to call it a comprehensive "solution".

We need game engines to:

1. Do all they can to make collecting and precaching PSOs as painless and seamless as possible for developers
   • Automatic PSO gathering and precaching to catch most PSOs without effort
   • Better UX/UI for manually collecting PSOs and bundling them
2. Better profiling tools for identifying origins of stutters
3. More developer awareness efforts and better signposting within the engine
   • More streams like the ones Epic just had for PSOs to raise awareness
   • Streams going through the workflow of how to properly manually collect PSOs, bundle them, and test your game
   • In editor reminders of how important it is to ensure PSOs are collected and precompiled, and alert when game is running with warmed cache vs a cold cache

We need game developers/publishers to:

1. Take stuttering seriously and do everything they can to mitigate/minimize the issue
   • There needs to be an actual solid effort to collect and precompile PSOs/shaders, not just a surface level attempt to pull the wool over gamers' eyes with a "compilation screen"
   • It's a tedious, but necessary part of development. This is directly tied to user experience.. the most important aspect of any game!
   • There's no excuses.. it's not an aspect of development to cut corners on right at the very end!
2. Be open and responsive to feedback regarding stuttering
   • Often this word is avoided unless enough people scream it, and developers often try to dance around the issue by saying "some stutters on certain configurations..."
   • Engage with the community and be honest about what the intentions are regarding fixes and improvements

We need Microsoft to:

1. Provide a Fossilize like solution for DX12 which allows platform holders like Steam/MS/Epic to utilize and host shader precaching downloads/uploads for games
   • Allows gamers to download gathered caches and compile them locally before the game is started
   • Allows gamers to crowdsource caches for games which have already released.
2. Provide a DX9-11 (or at the very least DX11) to DX12 translation layer so most games can utilize the new caching/precompiling system
   • Same as above... allows gamers to create caches for games which have already released and missed out on a shader precompilation process
3. Improve the APIs
   • Do everything they can to ensure devs can minimize the amount of PSOs/shaders generated through the API and reduce compilation times (thinking something like VK graphics-pipeline-library)

We need IHVs to:

1. Ensure drivers are updated and optimized for games to reduce stuttering where possible
   • Sometimes running an unoptimized shader at a higher cost to execute is better than stuttering to generate the more optimal code in the first place
2. Cut support for very old hardware
   • Old hardware holding API advancements back due to lack of feature support

We need Gamers to:

1. Respectfully call out this issue and be honest about it existing and affecting games
   • Bring attention to the issue and speak out about it on forums, social media, and in the proper tech channels to generate awareness/discussion
   • Please stop going into threads and denying this issues exists simply because you never noticed it or it didn't happen to you!
   • If you do not have this particular issue with the game, simply enjoy it.. don't ruin others' efforts to improve the situation
2. Respectfully report issues to developers
   • Hit the game's official forums and post your experience without being condescending or rude
   • Use the official feedback channels and create tickets showing with clear evidence where it's happening and how it's repeatable
3. Understand that it's impossible to be perfect in an industry with so many moving parts
   • We know PC development is more difficult than consoles with many more variables and expectations should be realistic
   • We should not expect perfection, but we should expect high quality
   • The pursuit of perfection is worthy of attention and high praise, and well optimized games which release with little to no stuttering should be called out and exalted!
4. Keep publishers honest
   • Continually call them out for releasing products which stutter and don't perform well at launch
   • Don't pre-order and buy games with these issues evident
5. Call out and reward the developers who continually engage with their communities and improve their games over time. Sometimes things don't work out as planned right at launch, but there's something to be said for developers which continually work to make their games better and don't just drop them

I think you need something like that, which tackles the problem on multiple fronts to have a more comprehensive "solution". If Epic's work eliminates 90% of the issue for developers, that's great and amazing for their engine and devs which use it.. It helps reduce the amount of manual work.. but it's it's not a solution for the games which have already released.. or games which are on other engines, or development in general. There's a lot more that can be done than is what's currently being done, so we need more commitment from others within the industry as well. But at this point, I'm happy that Epic has taken the feedback seriously and are at least attempting to make the lives of devs which use their engine easier. IMO, Epic has moved the ball back in the court of the developers.. where again we need to keep pressure to ensuring that they're also doing what they can to minimize this issue on their end. FF7 Rebirth is just an obvious example of a developer that wants to appear like they've attempted to do what they could... but in reality just did the bare minimum. We can't be fooled by games which have a shader precompilation screen at start up and assume they've done the job properly.

A multi-pronged like stated above approach would catch and hopefully mitigate these issues.

 

[digitalwanderer]

Dumb question: why not do the shader compiling as the end of the installation process instead of the beginning of the gaming experience? Seems less likely to draw as much attention to itself.

 

[Remij]

Dumb question: why not do the shader compiling as the end of the installation process instead of the beginning of the gaming experience? Seems less likely to draw as much attention to itself.

You're absolutely right that it would draw way less attention to itself if it were done within the download/install process. But typically the game code has to execute and thus be running before it could pre-compile its shaders, and so obviously the game has to be downloaded and installed first. Fossilize for Vulkan avoids this requirement because has everything it needs to precompile PSOs without even requiring the game.. and Steam has integrated it into the download/install process.

For Valve to be able to do something similar with Windows and DirectX games currently, it would have to download/install the game, then automatically run a batched pre-compilation process made by the developers in the background and show on Steam's UI that the game is compiling shaders. Then when the process was done, it could alert the player that the game is ready to play.

 

[tuna]

Alex is on Bsky now, has a thread on a recent Sony State of Play and the preponderance of UE5 titles.

While UE's issues extend beyond just gathering PSO's, I'm sticking this in this thread as the point about it being improved/solved in the latest version of UE5 means that we will still see many games with missed PSO's in the coming years as it's just not tenable to switch to the latest version of an engine at a certain point in development.

Why not? That is really old style thinking, you should really be able to move to a later UE version very late in development of your game.

 

[Rodéric]

Dumb question: why not do the shader compiling as the end of the installation process instead of the beginning of the gaming experience? Seems less likely to draw as much attention to itself.

That's what we did on Total War Shogun 2, however to benefit from drivers enhancements we also recompiled them when drivers changed at game startup...

 

[Scott_Arm]

Do new drivers always have to recompile shaders, or is there some way for the shader compiler to know that a shader is recent enough that it will still work? I'm guessing there's nothing in a file signature or part of the compiled binary that can tell it which compiler version was used? Actually, there must be otherwise it would have to recompile shaders every time. Or do the games just check driver versions and force recompile? The driver shader cache might have some meta data that handles this, and it's not baked into the binaries?

 

[DegustatoR]

Or do the games just check driver versions and force recompile?

Yes. Compiler can change between driver versions and it's not guaranteed that a binary compiled for a previous driver would work without issues - or that such changes weren't in fact made to solve some previously disovered issue.

 

[DavidGraham]

Do new drivers always have to recompile shaders

Yes they do, different game versions (via patches) will also recompile, changing graphics settings could also force a recompile.

 

[Rodéric]

Do new drivers always have to recompile shaders, or is there some way for the shader compiler to know that a shader is recent enough that it will still work? I'm guessing there's nothing in a file signature or part of the compiled binary that can tell it which compiler version was used? Actually, there must be otherwise it would have to recompile shaders every time. Or do the games just check driver versions and force recompile? The driver shader cache might have some meta data that handles this, and it's not baked into the binaries?

With Vulkan, IF the IHV does a correct job, there's a version number you can check to see whether you need to recompile or not.
I think it's in VK_KHR_pipeline_binary, but don't quote me on that, I haven't used it yet.

On Shogun 2, we checked the driver version and recompiled when it changed.
As for Windows' shader cache... I consider it a fail safe for poor programming... Something you should ignore, as you should be handling that in your application.