Shader Complilation on PC: About to become a bigger bottleneck?

I was kind of expecting halo infinite to support directstorage but I guess not

 

1. XBO game, doesn't need it since it can't really take advantage of it.
2. It's not out of beta on PC yet. And probably won't be till the end of 2022 now.

 

I'm really glad that @Dictator brought this up. It ABSOLUTELY needs more attention brought to it. There are a number of ways to mitigate this issue, and what we need are people like Digital Foundry and other outlets was well as developers themselves to champion them so they become general practice across the industry!

• Performance profiling and QA should be catching these stutters and developers should be optimizing and eliminating them before the game ever ships, not after the fact.
  • This is basic quality assurance... you're making a product and it should work without freezing and hitching.
  • If your game can't properly compile shaders at run-time causing stutters and degraded performance, you should be pre-compiling them.
• Allowing the user the option to Precompile shaders is a must going forward!
  • All games should either do it as standard, or at least give the user the option to create a "full" shader cache.
  • The mechanism exists to precompile shaders/PSOs after initial load to reduce/eliminate this issue.. we need devs to understand that it's OK to implement this process at initial boot, or give the user the option to build/rebuild the cache whenever. (Like Call of Duty for example)
  • Shaders are precompiled on console for a reason... Doing so reduces/eliminates stutters, improves CPU performance, and also reduces loading times.
• Nvidia, AMD, Intel, and Microsoft need to do whatever it is they can to fix this issue.
  • Hey Nvidia/AMD/Intel, I'll let you in on a secret... I don't give a crap if you have 10 more FPS than your competitor.. if your card has stutters and theirs doesn't, and that becomes a trend... I'm jumping ship instantly. What I'm saying here is sometimes, the effort to "improve benchmark scores" can come at the cost of stability and predictability of the drivers, causing issues for developers when debugging and profiling their games. STABLE performance comes before HIGHEST performance... always. Optimize to eliminate stutters and hitches before optimizing to increase FPS!
  • MS can always be improving Windows and DirectX. Getting developers those features they're asking for, as well as bringing all the IHVs together to set standards and continuing to push the industry forward. Hopefully with their renewed and recent success in the PC market (their games often dominate Steam's Top 10 sellers lists when they release)

I believe precompiling the shaders is the obvious route to go forward. On mid/high end PCs, waiting a few minutes upon initial boot for the shaders to compile is a non issue to me. Hitches and stutters during gameplay are FAR worse, and completely unacceptable. The problem is that on some PC configurations this process can take a prohibitive amount of time.. and everyone's tolerances are different when it comes to these sorts of things. That's completely understandable.. and I think a good solution to that problem would be that when you run the game the first time (or the game detects an invalid shader cache) it connects you to a server which collects the necessary information about your GPU and driver version, and then throws a bunch of CPU cores at compiling your shader cache very quickly... dropping a 5-10 minute compilation process down to 1-2 minutes. If you aren't connected to the internet, then you simply have to wait the full duration as your processor does the compiling. This process will become faster over time and would scale with CPU cores.. so in the future, when you upgrade, your compile times will become shorter and shorter.

Another think I think which would go a long way into reducing the user experience impact of this "compilation process" is if Steam and other storefronts integrate this compilation step, into the "download/install" process. Perhaps for example Steam could download the game, and after that, it automatically runs the game in the background and begins the shader compilation process. To the user, it would appear that Steam is still "downloading/installing" the game. Then once the compilation process is done, Steam closes the game down and says your game is ready to play. The user runs the game and was never the wiser since there was no "compilation process" they had to endure after Steam told them their game was "ready to play".

Also developers who have compilation processes at initial load, please don't just leave the screen black.. or put a little logo with no other information. If you're pre-compiling shaders, the best thing you can do is actually state that's what you're doing and either have a progress bar, or state that it will take some time. I can't count the amount of times I've seen people immediately jump on the forums and blast a game for "infinite loading" "stuck at loading screen/black scree" when the game was just precompiling shaders... You need to give the user that information so they don't needlessly think your game is busted.

And Alex, PLEASE do a video on this subject. It needs a full in depth video. Please go into the problem itself from the developer perspective, various mechanisms to mitigate the problem, the reasons why precompiling processes are a good thing (performance, stability, loading times) despite the obvious drawbacks... as various solutions to reduce those drawbacks such as what I stated above. It's absolutely needed. I've seen enough shout outs of Digital Foundry in various GDC tech talks and twitter posts to know that you guys have a positive effect and influence on many developers in the industry. What is needed is awareness and knowledge so that users can PROPERLY report constructive feedback to developers themselves.

 

If I am not wrong (probably I am) shaders are compiled considering CPU+GPU+VRAM+RAM combinations.

It could be doable:
- Check hardware.
- Install game (most probably download).
- Upload game Id and hardware info to cloud.
- Search for precompiled shaders.
---- If, shaders exits download.
---- If not, compile and download.

I think an R5 5600 + GTX 3060 and an R5 5600X + GTX 3060 could use the same compiled shader, and in case not, once a combination is compiled the job is done.
What I do not know is "how big" compiled shaders are, so it is practical to download them.

 

Upload where?
I feel like any sort of solution which would involve uploading and downloading won't work well here.
Solution should be local, like the ability to run shader compilation after game's installation in the background by requesting this through a standard OS interface.
Then again it's hardly much better than just doing shader precompilation at a game's start. Especially since you'll still need to do that in case of driver and/or OS updates.

 

I really do not know. Microsoft, Game editor, Portals (Steam, Epic)???
I think that is more a political than technical question ¿Who does the work?

Games are constantly patched, and uploading GPU + driver revision + game version (thanks Flappy Pannus) is no so difficult.
As patching your game with the recompiled shaders.
Compiled really fast in a server once and server thousands of times.

 

It is both a political and technical question which can have rather long lasting repercussions like who is able to provide a better gaming experience and will your games even work without an internet connection now.
We all should be careful in suggesting such things because ultimately they can make the whole market a lot worse than it is now, with this issue of hitching in some games.

Games are constantly patched by the developers (who don't have server farms), publishers (who generally don't care about games outside of their launch window sales) and these patches are delivered by a whole bunch of content distribution platforms.
The latter ones are the most "neutral" candidates here but they would have to work with both ISVs and GPU IHVs to make something like that work, and we may end up in a situation where just one IHV is able to provide this data for example, with many ISVs just ignoring the option - just like they do now. I mean it's not like it's impossible to remedy the problem right now, without a complex server side solution.

 

It's really in Nvidia/AMD's best interest to put a large effort into "solving" this problem. It has to get better, because at some point, it will get bad enough where a person asks themselves why am I spending this amount of money on the high end hardware and dealing with these issues when cheaper consoles are able to perform smoothly in these specific situations?

People always kind of looked the other way, or the uninformed would just assume there was some other issue with their PC causing games to hitch or stutter... and there was always some who would claim some "fix" would work and they'd no longer have any stutter... not realizing that it was simply the fact that the shaders had already been cached by that point. And then there are the people who can't accept any faults with their preferred platform and act like as if it's always some user error and that it's not the games fault or the platforms fault. But I think now things are starting to change. Tech channels like Digital Foundry and others are starting to clue people in to some of these broader issues, and it's important for people to realize that although there are some drawbacks, there IS a path forward to massively reduce/eliminate this issue. People just have to understand why it's a necessary evil.

And yea, lol, a friend of mine actually refunded Persona 5 Strikers because of the shader compilation process at initial launch. He downloaded it, started it up and it's just a black screen with the logo in the corner. He's said "the damn game was just frozen on the loading screen for 10 mintues, I just shut it off... I'm not dealing with that". He's got an older processor so it takes some time on his PC. After he told me I convinced him to buy it again and just wait it out because it wasn't broken it was just doing the shader comp process. So he did and waited a bit and sure enough it started up, and then he's like "Oh the game is great, runs beautifully too. Doesn't take long to load at the start any more either".. haha

So yea, that's why I think it's really important for developers to be clear as possible with what the game is doing. Some people don't have much patience anymore these days. =P

 

Yea the best way to do it is having the process done locally upon initial boot and drivers/patches. On a modern high end CPU, compilation times aren't really bad at all. I think 5-10 minutes is reasonable if required. And I think getting Steam/Epic involved in "hiding" that process by incorporating that step into the "download/install" process, it would go a long way to mitigate the issue without much drawbacks.

As people upgrade their CPUs and with higher core counts, compilation times come down drastically, so it's something that can somewhat scale as people upgrade too!


It's just the thought of being able to offload the process to a server somewhere that could crunch through it extremely quickly would be a very nice thing for people on the lower end of the game's requirements.

I am happy to see more and more games doing the precompilation process at first boot these days though.

 

Personally I think that this can be excessive.
Latest COD titles are doing it properly IMO - they do precompilation while a user is setting up the game after launch.
It doesn't work on driver changes of course but you're not forced to wait till it completes either.
Horizon is an even better example where you can just skip it and launch the game anyway and it will finish in the background. This IMO is the better way of doing this.
And before anyone says that background compilation will lead to hitching - no, it won't, when done properly. There are a multitude of games which don't hitch with such background precompilation running.

 

The idea of incorporating it into the download and install process is great because potentially that could be engineered to happen CONCURRENTLY to the download. The store-front can prioritize downloading the shader data first (through developer set flags) so it can start compiling shaders while the rest of the game data is downloading. Just make sure to pause that process if the GPU/CPU starts getting used heavily. No point in pre-compliling shaders to avoid stutter in one game, if the precompilation is happening while the user is playing another game and getting stutters THERE. In fact, imagining that scenario is almost hilarious.

 

I'm very happy to report that Double Fine has just recently patched Psychonauts 2 (Unreal Engine 4) and those compilations stutters are GONE!

Update 1089619 addressed the following issues:

• Fix intermittent hitches on PC.
  
• Fix water in Bob's Bottles displays incorrectly in photo mode.
  
• Fix lighting issues in Strike City and Cassie's Collection
  
• Fix dialog tree delay in Fatherland Follies.
  
• Fix Panic Attack not using its phantom attack during Psychoseismometer battles.

Used my completed save file and played a bunch of areas and did a bunch of things that I knew for a fact would stutter in the old version, and perfectly smooth 150+fps at all times. Great work from the team! A bit late, and it shouldn't have released in that state, but they took the time and fixed whatever the problem was. I'm very happy because it's such a good game! It deserves to be the best version it can be.

 

I dont know what unity done but shader compiling is magnitudes faster now for me

One shader and its permeations
unity 2021.1.15 time > 1:00 i.e. over a minute
unity 2021.2.3 time < 0:01 i.e. less than a second

 

I've beaten the game back in Sep and haven't really seen much stutters in it. Granted I've played in 60Hz, maybe they've appeared only in higher refresh rates?