Shader Compilation on PC: About to become a bigger bottleneck?

[Remij]

I just dont update my drivers that often. I also usually wait to play the latest games (not necessarily deliberately, just because I always have a massive backlog of older games) so the initial patching frenzy is over.

I've no problem at all with an initial shader compile. I just see it as part of the download/install wait. I do agree repeated compiles after that are ennoying though which is why I avoid driver updates unless I'm starting a new game that isn't optimised in my driver. But in that instance I generally update the driver before installing the game.

Me too brotha. I will wait however long it takes to be able to experience the game the way it should be. I look at is as part of the "install" process as well. Like I said, I think it's more about the mentality or perception. We wait for things to download and accept it's a fact of life. Some people take 5 hours to download a 70GB game.. others 10 minutes.. So I think if there was a way to get the process out of the way before the user opens the game for the first time, regardless of how long it would take, then it would more accepted. Like why can't these shader processes just be batch files which are run automatically by Steam after the download is done?

As for drivers, yea.. not a big deal to me either, but it would be nice if it was possible for drivers to essentially be "portable".. I guess, if that's the right way to describe it. Like basically, you could drop some driver files in the directory of the game and the GPU would always use those drivers for that application, and if it didn't find any it would fallback to default one.. or something like that.

But I completely understand people who are fed up with it and just don't want to deal with this stuff.. but anyway, the industry is groaning and things are happening so we'll see what they come up with and the direction things go, now that it's a hot topic.

 

[Scott_Arm]

It would be nice if there was a way to include the first shader compilation in the install process so it wouldn’t happen the first time you load.

 

[Scott_Arm]

Why do shaders have to recompile after every driver update? I can’t see why that’d have to happen.

 

[Remij]

Why do shaders have to recompile after every driver update? I can’t see why that’d have to happen.

They don't have to always happen.. but when you have a game like TLOU P1, where the developers are actively working on aspects of the game which directly affect that, then yea you're going to have to rebuild shaders.

 

[DavidGraham]

Several new games require you to update your driver to gain access to DLSS2/DLSS3 or ray tracing, or to run DLSS/RT with better quality/performance, or even to fix some bugs that plague other games (flickering/missing textures/crashes ..etc). You also need new drivers to access newly added features, play early access titles or betas .. etc.

Often times, you invoke the compilation process twice, once with the new driver, then with a new game patch. You have to play the synchronization game, where you synchronize the driver update with the game update to avoid the duplicate compilation.

They don't have to always happen.

They always do happen unfortunately.

Why do shaders have to recompile after every driver update? I can’t see why that’d have to happen.

Each driver has different behavior and performance regarding these awful PSOs, so a driver update ALWAYS invalidates the old shader cache, and invokes the rebuilding of a new one. Worse yet, some games rebuild shaders if you changed some visual settings from High to Low or vice versa, even if the driver and game version stayed the same.

It's a mess.

 

[Remij]

They always do happen unfortunately.


Each driver has different behavior and performance regarding these awful PSOs, so a driver update ALWAYS invalidates the old shader cache, and invoke ls the rebuilding of a new one. Wrose yet, some games rebuild shaders if you changed some visual settings from High to Low or vice versa.

It's a mess.

No they don't. The latest driver update from Nvidia didn't require me to precompile all my shaders again in TLOU P1. It started at 98%.. and only a few were invalidated.

And the only games that do that require recompilation after changing settings... are the ones that don't pre-compile shaders for all the various settings upfront like most do. And in those instances, the wait would be far shorter than before.

Like I said, just get a console now. You'll never be happy.

 

[DavidGraham]

The latest driver update from Nvidia didn't require me to precompile all my shaders again in TLOU P1.

Was that the Hotfix? Hotfixes usually don't require rebuilding. As they are essentially the same previous version just with added fixes.

Like I said, just get a console now. You'll never be happy.

Never, I am a PC guy through and through.

 

[Remij]

Was that the Hotfix? Hotfixes usually don't require rebuilding. As they are essentially the same previous version just with added fixes.


Never, I am a PC guy through and through.

Sure. So in other words I'm not getting driver updates "all the time".. and it's entirely up to me to decide when I want to update my drivers.

Well that's good... so then come back to earth for a second. You know this is a reality of the openness of the hardware. I agree 100% with you that it sucks... but as it currently stands, we have to accept that there will be these compilation processes. Developers can do what they can to minimize the impact.. and I think there's other ways the industry could help alleviate the friction felt by PC gamers.. but we're still at the point of getting developers to incorporate this step in the first place. Once it becomes standard.. then hopefully we can look at ways to make it less annoying for players.

I'm just thankful on PC we can at least multi-task while it's doing its thing. Use that shader compilation process time to hop on your favorite technology forum and write up a good rant about it, for example

 

[DavidGraham]

Sure. So in other words I'm not getting driver updates "all the time".. and it's entirely up to me to decide when I want to update my drivers.

I get that, but if you wanna stay on the edge, and up to date with new tech, new features new games you have to upgrade. And Like I said, each new "major" driver invalidates the shaders, anyone here plays any of the Forza Horizon titles? Or even any of the regular Forza games? These games are notorious for this kind of stuff. I wanna hop in quickly to fly on the road in my favorite racing game, then I get hit by a 13 minutes shader building process, and I am bummed instantly.

Other notorious titles include Detroit, Horizon Zero Dawn and all of the recent Call of Duty games.

then hopefully we can look at ways to make it less annoying for players.

I really hope a solution comes fast.

Use that shader compilation process time to hop on your favorite technology forum and write up a good rant about it, for example

Yeah man, lol, I could do that.

 

[Remij]

I get that, but if you wanna stay on the edge, and up to date with new tech, new features new games you have to upgrade. And Like I said, each new "major" driver invalidates the shaders, anyone here plays any of the Forza Horizon titles? Or even any of the regular Forza games? These games are notorious for this kind of stuff. I wanna hop in quickly to fly on the road in my favorite racing game, then I get hit by a 13 minutes shader building process, and I am bummed instantly.

Other notorious titles include Detroit, Horizon Zero Dawn and all of the recent Call of Duty games.

I really hope a solution comes fast.

Yeah man, lol, I could do that.

I know, I get your pain for sure.. but we tend to overexaggerate these things when were pissed off. I certainly don't feel like I'm always updating drivers.. but yes, it can be annoying when you have, and then come back to a game and it needs to recompile.

Forza Horzion compiles in less than a minute for me... not really a big issue at all personally, but I can understand people on 6 core/thread CPUs getting annoyed. Detroit and Horizon were both console games developed without ever considering that they might come to PC in the future, so again, we have a lengthy compile time as there's so many.

I wouldn't expect a solution to come fast per-se, but as more games have these lengthy processes, people will inevitably complain, and I think eventually you'll see companies do what they can to mask them as best as possible. I like the idea of integrating them into the download/install cycle. They'll still be there, no doubt, but they could be less obvious. Compilation times usually scale with CPU cores, so in the future, faster CPUs with more cores should help compile these games faster, and given the changes potentially happening in the API space, there could be ways for developers to minimize the sheer amount of PSOs and permutations in the first place. Now that Sony and MS both release games on PC, perhaps they'll have this in mind from the start since future games could end up on PC at some point.

 

[Remij]

Also, it sounds like there's a ton of discussion/arguing going on about pipelines and dynamic states right now.. so the industry is definitely buzzing.

 

[Ethatron]

Opinions are clearly divided on this but why are some graphics developers so obsessed with explicit control over things that they can never control unless they start writing GPU assembly? I get why DX11 can be limiting with the high level of synchronization and babysitting required in the driver. With DX12 and Vulkan developers have far more explicit control of memory allocation and locks. Why is it a problem that the hardware and driver does further optimization? Seems to be driven by ego more than any practical reason.

I think one premise was, that at least from the API perspective, we could get from soft-realtime to maybe firm-realtime if not hard-realtime. The community increasingly creates applications which are critical (as in: hickups might hurt). And the APIs are for everyone to use, there's no distinct API for each realtime guarantee requirement.
I appreciate the predictability of behaviour with the (then) new paradigm. Our (team's) expectation from an ideal industry perspective was that some companies specialize on this and solve it and sell the solution to the industry in the long run. And we won't see a bit of it. But apparently sharing and collaborating is really difficult, especially if money is involved.
I think too it's a problem that IHV(s) want to have a say in the mentioned dynamic optimization step. Ideally non-IHV programmers should have the flexibility to do precisely what an IHV is magically conducting in the driver. Ideally this solution wouldn't be provided by every ISV on it's own, but by dedicated solutions.
The whole API mess is a mess because the controlling corporations make messy decisions for messy goals, not because of (e.g.) game programmers or even the driver programmers. Doing things in some way which is conceptually and generally good might require doing other things in some specific way, but if that way is inaccessible/opaque then the first thing is cockblocked. Sometimes there's not enough commitment to go through with it all the way, on the side of IHVs and ISVs.

 

[trinibwoy]

I think one premise was, that at least from the API perspective, we could get from soft-realtime to maybe firm-realtime if not hard-realtime. The community increasingly creates applications which are critical (as in: hickups might hurt). And the APIs are for everyone to use, there's no distinct API for each realtime guarantee requirement.
I appreciate the predictability of behaviour with the (then) new paradigm. Our (team's) expectation from an ideal industry perspective was that some companies specialize on this and solve it and sell the solution to the industry in the long run. And we won't see a bit of it. But apparently sharing and collaborating is really difficult, especially if money is involved.
I think too it's a problem that IHV(s) want to have a say in the mentioned dynamic optimization step. Ideally non-IHV programmers should have the flexibility to do precisely what an IHV is magically conducting in the driver. Ideally this solution wouldn't be provided by every ISV on it's own, but by dedicated solutions.
The whole API mess is a mess because the controlling corporations make messy decisions for messy goals, not because of (e.g.) game programmers or even the driver programmers. Doing things in some way which is conceptually and generally good might require doing other things in some specific way, but if that way is inaccessible/opaque then the first thing is cockblocked. Sometimes there's not enough commitment to go through with it all the way, on the side of IHVs and ISVs.

What does soft/firm/hard realtime mean?

Theoretically Nvidia, AMD and Intel could treat their ecosystems like consoles and expose IHV specific apis that ISV's can target. That doesn't seem like a good solution though as instead of having to build 3 renderers for PC/Xbox/PS you now have to build 5 for Intel/AMD/Nvidia/Xbox/PS. But then as long as you have that "PC" abstraction layer to deal with the PC IHVs will want to optimize for their hardware which is perfectly reasonable.

 

[trinibwoy]

Good discussion on DX12 PSO's and #stutterstruggle in Nixxes' Spiderman GDC talk.

Advanced Graphics Summit: 'Marvel's Spider-Man' Remastered: A PC Postmortem

Marvels Spider-Man was originally developed by Insomniac Games in partnership with Sony Interactive Entertainment and Marvel for PlayStation 4 in 2018, followed by the PlayStation 5 version Marvel's Spider-Man Remastered in 2020. Marvel's...

gdcvault.com

 

[DavidGraham]

Good discussion on DX12 PSO's and #stutterstruggle in Nixxes' Spiderman GDC talk.

Sigh, is that's why Spider-Man is so CPU intensive? They've tied PSOs to material streaming, as the game streams asset by asset, each PSO for that is asset is compiled in real time in separate CPU threads.

 

[trinibwoy]

Sigh, is that's why Spider-Man is so CPU intensive? They've tied PSOs to material streaming, as the game streams asset by asset, each PSO for that is asset is compiled in real time in separate CPU threads.

It shouldn't as the material streaming and PSO building happens on loading screens and not during gameplay. The high CPU usage is due to BLAS/TLAS construction.

 

[pjbliverpool]

Really interesting talk. I find the PSO choices a little questionable though. It seems they intentionally disregarded a pre-compilation stage on first run in favour of doing this in the load screens instead. Which means instead of 1 long wait (they mentioned "a few minutes") at the start, we get consistently slower loading times throughout the game. That sounds like not a great compromise to me. They do also mention that it can be done while streaming too during gameplay, I assume that only happens if something has been missed by the load time process for some reason, but may still contribute to higher CPU requirements a little. Although I assume that would happen if you had an initial pre-compilation stage as well.

The fact that they constructed their own memory management system as well rather than relying on vidmm is great. I suspect many games do not do this which is why they have more severe issues when they run out of VRAM. Given they created 2 extra threads just for this memory management though (unique to PC with it's non-unified memory) that is likely to add some additional CPU overhead. It was also interesting to hear that although the utilise the capability in PC's for the GPU to access system memory, they do not yet utilise the much newer reverse capability. So there is presumably scope for further improvement there in their memory management.

It was clear from the video how experienced and dedicated they are to PC porting though. It's obvious why they are so highly regarded.

 

[Remij]

Really interesting talk. I find the PSO choices a little questionable though. It seems they intentionally disregarded a pre-compilation stage on first run in favour of doing this in the load screens instead. Which means instead of 1 long wait (they mentioned "a few minutes") at the start, we get consistently slower loading times throughout the game. That sounds like not a great compromise to me. They do also mention that it can be done while streaming too during gameplay, I assume that only happens if something has been missed by the load time process for some reason, but may still contribute to higher CPU requirements a little. Although I assume that would happen if you had an initial pre-compilation stage as well.

The fact that they constructed their own memory management system as well rather than relying on vidmm is great. I suspect many games do not do this which is why they have more severe issues when they run out of VRAM. Given they created 2 extra threads just for this memory management though (unique to PC with it's non-unified memory) that is likely to add some additional CPU overhead. It was also interesting to hear that although the utilise the capability in PC's for the GPU to access system memory, they do not yet utilise the much newer reverse capability. So there is presumably scope for further improvement there in their memory management.

It was clear from the video how experienced and dedicated they are to PC porting though. It's obvious why they are so highly regarded.

I dunno, I don't remember Spider-Man: Remastered having any egregious load times. They're a bit longer than PS5.. but probably worth it the way they did it instead of a full upfront compile of all PSOs. I think Nixxes has the right idea though. They said a pre-compilation stage would have been around 3 or so minutes... that's honestly nothing compared to TLOU P1 or Uncharted.. lol. Sounds like Insomniac like to keep their materials and shaders lean, which bodes well for potential future ports, especially if they're now keeping PC in mind from the beginning. Insomniac certainly are a productive studio.. it makes sense as to why.

I think having a short 1-2min pre-compilation stage to get the real heavy ones out of the way, then compile what you must during any loading screens, or in the background when streaming is a good idea. Basically keeping the "upfront" compile to a minimum, while also not having the loading screens take too long.

Honestly though, you cant go wrong with giving the user the option. Have it like I mentioned above as default, and if the player wants to wait and compile everything upfront they can go into the settings and select to do so.

 

[milk]

It seems they intentionally disregarded a pre-compilation stage on first run in favour of doing this in the load screens instead. Which means instead of 1 long wait (they mentioned "a few minutes") at the start, we get consistently slower loading times throughout the game.

I just don't understand why it can't be both. It should be compiling shaders any time the engine sees some spare CPU cycles, and keep track of that so when streaming the ones already compiled can be skipped. In fact, I'd include the compilation of the very first shaders (the ones used on intro screens and menus) as a hidden part of the game instalation itself, for maximum presentation polish on first launch.

 

[Remij]

I just don't understand why it can't be both. It should be compiling shaders any time the engine sees some spare CPU cycles, and keep track of that so when streaming the ones already compiled can be skipped. In fact, I'd include the compilation of the very first shaders (the ones used on intro screens and menus) as a hidden part of the game instalation itself, for maximum presentation polish on first launch.

Essentially this is how I'd like to see this stuff implemented in the future. I'll use Steam as an example, but it could apply to any launcher or application itself.

• User purchases a game on Steam
• Begins downloading a game, time passes...
• Download finishes and Steam automatically kicks off a batch file which begins the shader compilation process in the background (think of it as just another "pre-requisite" required to install)
• Progress of pre-compilation is detailed in the downloads tab
• Pre-compilation finishes and automatically closes
• Game is *actually* ready to play when you click play

I honestly think this is the best way to go about it! It's an unfortunate reality that we have to do this on our own machines... but the goal should be to make it as frictionless as possible for gamers. I think perhaps removing the large initial pre-comp process from being "inside the game" and moving it instead to the "download/install/decrypt/compile" stage where perhaps it more suitably belongs, will be much better. People know and understand that things aren't ready yet when Steam is still downloading/decrypting/installing... but when they finally get into the game.. they expect the game to be ready. I think doing it this way would help a lot. Because I know for myself personally, if a game is downloading, installing or decrypting... I'm on the net messing around or listening to a video or something while I'm waiting, and you don't feel like you're just wasting time as much. When a user gets into a game, they expect to be able to play right away.. that's the way it should be.