Shader Compilation on PC: About to become a bigger bottleneck?
- Thread starter Flappy Pannus
- Start date
DegustatoR
Legend
1. XBO game, doesn't need it since it can't really take advantage of it.
2. It's not out of beta on PC yet. And probably won't be till the end of 2022 now.
Flappy Pannus
Veteran
Alex from DF brought this up on today's DF Direct Weekly, at least with respect to Unreal Engine 4 and it's shader compile stutters in DX12 wrt Hellblade, and mentioned he may do a separate video on it. Obviously I'm all for that, but I would suggest it would be expanded to talk about the shader compile issues on PC in general (as Richard alludes to this is a bigger issue than UE4 games), which Alex does go into some detail as well and mentions current attempts to mitigate this, such as Steam's sharable cache as one example (which I find works extremely well, in fact it's part of my purchasing decision now for those games when they're offered on other storefronts) - albeit he doesn't mention it's Vulkan/GL only (which as another poster in this thread indicated may at least be partly due to a fundamental architectural issue in DX12).
So yeah, that would be great to see - admittedly would be quite a bit of work though to cover everything as this thread has shown! DF has a large platform and could probably have an effect in getting more attention on this and at least making a 'compile all shaders' option in a game's settings menu as expected as a resolution slider in games doing forward. I'd like to more effort to perhaps mirror Steam's approach to sharable compiled caches that are digital storefront agnostic (as Alex mentions this may be the responsibility of the GPU manufacturers ultimately), but at the very least there's gotta be that option to do it upfront and not in-game if you so choose.
So yeah, that would be great to see - admittedly would be quite a bit of work though to cover everything as this thread has shown! DF has a large platform and could probably have an effect in getting more attention on this and at least making a 'compile all shaders' option in a game's settings menu as expected as a resolution slider in games doing forward. I'd like to more effort to perhaps mirror Steam's approach to sharable compiled caches that are digital storefront agnostic (as Alex mentions this may be the responsibility of the GPU manufacturers ultimately), but at the very least there's gotta be that option to do it upfront and not in-game if you so choose.
Last edited:
I'm really glad that @Dictator brought this up. It ABSOLUTELY needs more attention brought to it. There are a number of ways to mitigate this issue, and what we need are people like Digital Foundry and other outlets was well as developers themselves to champion them so they become general practice across the industry!
- Performance profiling and QA should be catching these stutters and developers should be optimizing and eliminating them before the game ever ships, not after the fact.
- This is basic quality assurance... you're making a product and it should work without freezing and hitching.
- If your game can't properly compile shaders at run-time causing stutters and degraded performance, you should be pre-compiling them.
- Allowing the user the option to Precompile shaders is a must going forward!
- All games should either do it as standard, or at least give the user the option to create a "full" shader cache.
- The mechanism exists to precompile shaders/PSOs after initial load to reduce/eliminate this issue.. we need devs to understand that it's OK to implement this process at initial boot, or give the user the option to build/rebuild the cache whenever. (Like Call of Duty for example)
- Shaders are precompiled on console for a reason... Doing so reduces/eliminates stutters, improves CPU performance, and also reduces loading times.
- Nvidia, AMD, Intel, and Microsoft need to do whatever it is they can to fix this issue.
- Hey Nvidia/AMD/Intel, I'll let you in on a secret... I don't give a crap if you have 10 more FPS than your competitor.. if your card has stutters and theirs doesn't, and that becomes a trend... I'm jumping ship instantly. What I'm saying here is sometimes, the effort to "improve benchmark scores" can come at the cost of stability and predictability of the drivers, causing issues for developers when debugging and profiling their games. STABLE performance comes before HIGHEST performance... always. Optimize to eliminate stutters and hitches before optimizing to increase FPS!
- MS can always be improving Windows and DirectX. Getting developers those features they're asking for, as well as bringing all the IHVs together to set standards and continuing to push the industry forward. Hopefully with their renewed and recent success in the PC market (their games often dominate Steam's Top 10 sellers lists when they release)
I believe precompiling the shaders is the obvious route to go forward. On mid/high end PCs, waiting a few minutes upon initial boot for the shaders to compile is a non issue to me. Hitches and stutters during gameplay are FAR worse, and completely unacceptable. The problem is that on some PC configurations this process can take a prohibitive amount of time.. and everyone's tolerances are different when it comes to these sorts of things. That's completely understandable.. and I think a good solution to that problem would be that when you run the game the first time (or the game detects an invalid shader cache) it connects you to a server which collects the necessary information about your GPU and driver version, and then throws a bunch of CPU cores at compiling your shader cache very quickly... dropping a 5-10 minute compilation process down to 1-2 minutes. If you aren't connected to the internet, then you simply have to wait the full duration as your processor does the compiling. This process will become faster over time and would scale with CPU cores.. so in the future, when you upgrade, your compile times will become shorter and shorter.
Another think I think which would go a long way into reducing the user experience impact of this "compilation process" is if Steam and other storefronts integrate this compilation step, into the "download/install" process. Perhaps for example Steam could download the game, and after that, it automatically runs the game in the background and begins the shader compilation process. To the user, it would appear that Steam is still "downloading/installing" the game. Then once the compilation process is done, Steam closes the game down and says your game is ready to play. The user runs the game and was never the wiser since there was no "compilation process" they had to endure after Steam told them their game was "ready to play".
Also developers who have compilation processes at initial load, please don't just leave the screen black.. or put a little logo with no other information. If you're pre-compiling shaders, the best thing you can do is actually state that's what you're doing and either have a progress bar, or state that it will take some time. I can't count the amount of times I've seen people immediately jump on the forums and blast a game for "infinite loading" "stuck at loading screen/black scree" when the game was just precompiling shaders... You need to give the user that information so they don't needlessly think your game is busted.
And Alex, PLEASE do a video on this subject. It needs a full in depth video. Please go into the problem itself from the developer perspective, various mechanisms to mitigate the problem, the reasons why precompiling processes are a good thing (performance, stability, loading times) despite the obvious drawbacks... as various solutions to reduce those drawbacks such as what I stated above. It's absolutely needed. I've seen enough shout outs of Digital Foundry in various GDC tech talks and twitter posts to know that you guys have a positive effect and influence on many developers in the industry. What is needed is awareness and knowledge so that users can PROPERLY report constructive feedback to developers themselves.
If I am not wrong (probably I am) shaders are compiled considering CPU+GPU+VRAM+RAM combinations.
It could be doable:
- Check hardware.
- Install game (most probably download).
- Upload game Id and hardware info to cloud.
- Search for precompiled shaders.
---- If, shaders exits download.
---- If not, compile and download.
I think an R5 5600 + GTX 3060 and an R5 5600X + GTX 3060 could use the same compiled shader, and in case not, once a combination is compiled the job is done.
What I do not know is "how big" compiled shaders are, so it is practical to download them.
It could be doable:
- Check hardware.
- Install game (most probably download).
- Upload game Id and hardware info to cloud.
- Search for precompiled shaders.
---- If, shaders exits download.
---- If not, compile and download.
I think an R5 5600 + GTX 3060 and an R5 5600X + GTX 3060 could use the same compiled shader, and in case not, once a combination is compiled the job is done.
What I do not know is "how big" compiled shaders are, so it is practical to download them.
DegustatoR
Legend
Upload where?
I feel like any sort of solution which would involve uploading and downloading won't work well here.
Solution should be local, like the ability to run shader compilation after game's installation in the background by requesting this through a standard OS interface.
Then again it's hardly much better than just doing shader precompilation at a game's start. Especially since you'll still need to do that in case of driver and/or OS updates.
Flappy Pannus
Veteran
Well on the technical level at least, it's certainly doable as Steam has shown - but for Vulkan/GL only atm. I've downloaded a new Nvidia driver an hour after launch and when I load up Steam it's already downloaded some (likely partial mind you) Vulkan/OpenGL compiled shaders for all games I have installed that use that API. It's extremely prompt.
But that's just Steam. Assuming the API is architected well enough to even facilitate this (as another posted indicated earlier this may be an issue with DX12 titles to even get these compiled shaders inserted), the problem is always going to be delivery in a multi-storefront world - who's responsibility is this? Microsoft of course has the network infrastructure to handle this, but would they want to take this on? Nvidia/AMD/Intel would probably be the ideal source of the compiled shaders (perhaps a driver feature where compiled shaders by gamers are automatically uploaded into the cloud that uses MS's network), I believe it would be in MS's best interest to facilitate another method to increase PC gaming's performance/accessibility but who knows if they would see it that way.
Of course no one is suggesting that be the only way compiled shaders are created, the option for every game to do some form of compiling before actual gameplay should always be there. If this thread has shown anything it's how incredibly complex this problem is, there won't be any single solution.
Flappy Pannus
Veteran
This is also a good idea, albeit it has to come after games even start to standardize on having the option to precompile all shaders before gameplay. On the backend this compiling only has to be done once per game per driver revision/GPU as well, for all players after that initial user upload they just immediately download the matching compiled shader cache for their driver, so not even a couple of minutes wait for most.
Yes. Dishonored 2 was notorious for this, especially when it shipped and people still had dual-core CPU's and were always asking "wtf why is it taking 5 minutes to load?" because it was a simple progress bar with no information. Just tell the user that you're compiling shaders and this has to be done only once per driver update.
Flappy Pannus
Veteran
Afaik, shaders are compiled against the GPU + driver revision + game version. Your CPU/RAM is not a factor in the resulting compatibility of the compiled cache, shader code runs on the GPU and doesn't care what model of CPU you have. A shader cache should not care if you have a Core i5 or AMD CPU.
I really do not know. Microsoft, Game editor, Portals (Steam, Epic)???
I think that is more a political than technical question ¿Who does the work?
Games are constantly patched, and uploading GPU + driver revision + game version (thanks Flappy Pannus) is no so difficult.
As patching your game with the recompiled shaders.
Compiled really fast in a server once and server thousands of times.
DegustatoR
Legend
It is both a political and technical question which can have rather long lasting repercussions like who is able to provide a better gaming experience and will your games even work without an internet connection now.
We all should be careful in suggesting such things because ultimately they can make the whole market a lot worse than it is now, with this issue of hitching in some games.
Games are constantly patched by the developers (who don't have server farms), publishers (who generally don't care about games outside of their launch window sales) and these patches are delivered by a whole bunch of content distribution platforms.
The latter ones are the most "neutral" candidates here but they would have to work with both ISVs and GPU IHVs to make something like that work, and we may end up in a situation where just one IHV is able to provide this data for example, with many ISVs just ignoring the option - just like they do now. I mean it's not like it's impossible to remedy the problem right now, without a complex server side solution.
It's really in Nvidia/AMD's best interest to put a large effort into "solving" this problem. It has to get better, because at some point, it will get bad enough where a person asks themselves why am I spending this amount of money on the high end hardware and dealing with these issues when cheaper consoles are able to perform smoothly in these specific situations?
People always kind of looked the other way, or the uninformed would just assume there was some other issue with their PC causing games to hitch or stutter... and there was always some who would claim some "fix" would work and they'd no longer have any stutter... not realizing that it was simply the fact that the shaders had already been cached by that point. And then there are the people who can't accept any faults with their preferred platform and act like as if it's always some user error and that it's not the games fault or the platforms fault. But I think now things are starting to change. Tech channels like Digital Foundry and others are starting to clue people in to some of these broader issues, and it's important for people to realize that although there are some drawbacks, there IS a path forward to massively reduce/eliminate this issue. People just have to understand why it's a necessary evil.
And yea, lol, a friend of mine actually refunded Persona 5 Strikers because of the shader compilation process at initial launch. He downloaded it, started it up and it's just a black screen with the logo in the corner. He's said "the damn game was just frozen on the loading screen for 10 mintues, I just shut it off... I'm not dealing with that". He's got an older processor so it takes some time on his PC. After he told me I convinced him to buy it again and just wait it out because it wasn't broken it was just doing the shader comp process. So he did and waited a bit and sure enough it started up, and then he's like "Oh the game is great, runs beautifully too. Doesn't take long to load at the start any more either".. haha
So yea, that's why I think it's really important for developers to be clear as possible with what the game is doing. Some people don't have much patience anymore these days. =P
Yea the best way to do it is having the process done locally upon initial boot and drivers/patches. On a modern high end CPU, compilation times aren't really bad at all. I think 5-10 minutes is reasonable if required. And I think getting Steam/Epic involved in "hiding" that process by incorporating that step into the "download/install" process, it would go a long way to mitigate the issue without much drawbacks.
As people upgrade their CPUs and with higher core counts, compilation times come down drastically, so it's something that can somewhat scale as people upgrade too!
It's just the thought of being able to offload the process to a server somewhere that could crunch through it extremely quickly would be a very nice thing for people on the lower end of the game's requirements.
I am happy to see more and more games doing the precompilation process at first boot these days though.
DegustatoR
Legend
Personally I think that this can be excessive.
Latest COD titles are doing it properly IMO - they do precompilation while a user is setting up the game after launch.
It doesn't work on driver changes of course but you're not forced to wait till it completes either.
Horizon is an even better example where you can just skip it and launch the game anyway and it will finish in the background. This IMO is the better way of doing this.
And before anyone says that background compilation will lead to hitching - no, it won't, when done properly. There are a multitude of games which don't hitch with such background precompilation running.
The idea of incorporating it into the download and install process is great because potentially that could be engineered to happen CONCURRENTLY to the download. The store-front can prioritize downloading the shader data first (through developer set flags) so it can start compiling shaders while the rest of the game data is downloading. Just make sure to pause that process if the GPU/CPU starts getting used heavily. No point in pre-compliling shaders to avoid stutter in one game, if the precompilation is happening while the user is playing another game and getting stutters THERE. In fact, imagining that scenario is almost hilarious.
Flappy Pannus
Veteran
Well Horizon might not fall under the category then at least on my system compared to other games that do it in the background - the initial load time alone to just resume your game is significantly increased, as is the stuttering if you don't start the game without at least 50%+ of your shader compiling stage done. I mean if you jump right in one area and run around for 5 minutes it might be ok, but fast travel to any other location and the loading time + hitching is awful. Granted though it's a pretty egregious case, most games don't use that many shaders and I 'only' have a 6 core CPU, I still appreciate they added the option so I may not have this experience when I upgrade my rig.
I agree that being able to skip the process as well as properly multithreading it so it's not crippling your fps should also be part of best practices though. Forced pre-game recompiles is not what I want either, but I want the option and what it's doing to be properly communicated to the player.
Yes as I noted in this thread early on, but many other games are also masking it during level loads, as well the amount of shaders has increased significantly in modern games. The expectation of extremely fast loading for modern games and at the same time, having more/longer shaders to compile reduces these opportunities to mask its impact on lesser CPU's.
Flappy Pannus
Veteran
Yes, that is a concern as well. Also bear in mind incorporating this into the install process does not change the fact that this compile has to be done for all installed games whenever you update your drivers. If this only had to be done once during the initial install process of a game and the shader cache was valid for the entirety of your installed game/OS I doubt I would have even been motivated to start this thread.
I'm very happy to report that Double Fine has just recently patched Psychonauts 2 (Unreal Engine 4) and those compilations stutters are GONE!
Update 1089619 addressed the following issues:
Used my completed save file and played a bunch of areas and did a bunch of things that I knew for a fact would stutter in the old version, and perfectly smooth 150+fps at all times. Great work from the team! A bit late, and it shouldn't have released in that state, but they took the time and fixed whatever the problem was. I'm very happy because it's such a good game! It deserves to be the best version it can be.
Update 1089619 addressed the following issues:
- Fix intermittent hitches on PC.
- Fix water in Bob's Bottles displays incorrectly in photo mode.
- Fix lighting issues in Strike City and Cassie's Collection
- Fix dialog tree delay in Fatherland Follies.
- Fix Panic Attack not using its phantom attack during Psychoseismometer battles.
Used my completed save file and played a bunch of areas and did a bunch of things that I knew for a fact would stutter in the old version, and perfectly smooth 150+fps at all times. Great work from the team! A bit late, and it shouldn't have released in that state, but they took the time and fixed whatever the problem was. I'm very happy because it's such a good game! It deserves to be the best version it can be.
Last edited:
DegustatoR
Legend
I've beaten the game back in Sep and haven't really seen much stutters in it. Granted I've played in 60Hz, maybe they've appeared only in higher refresh rates?
