Shader Compilation on PC: About to become a bigger bottleneck?

[Remij]

I've beaten the game back in Sep and haven't really seen much stutters in it. Granted I've played in 60Hz, maybe they've appeared only in higher refresh rates?

For me it was pretty bad. Especially in DX12.. every time hitting objects for the first time, or doing abilities... loading into new areas.. DX11 was a bit better.. but still not good.

Now DX12 is smooth as butter first time through doing anything. Wish it was like this the first time I played it.

 

[Remij]

Another thing I was thinking about is that tech sites/channels like Digital Foundry should add gaming "Shader Compilation" tests to their CPU benchmarks. They could pick a suite of games which have these pre-compilation processes and test how long they take to complete on various CPUs they are reviewing, as well as how well they scale with core counts.

I myself would honestly appreciate that, because if we can expect more of this stuff in the future, there's some value in knowing if it's worth shelling out a bit more money for more cores to drastically reduce the waiting time.

Also I just think it actually makes a very interesting comparison point between CPUs, which we haven't really seen much of.

 

[Remij]

@Dictator brought up an issue with Hellblade: Senua's Sacrifice's recent PC RT patch in a past DF Direct. The game suffered from stuttering using the DX12 path, but a patch has just been released which fixes those stutters by pre-generating shaders for both RT off and on presets.

Hellblade: Senua’s Sacrifice December 7th Patch Release Notes

• Fix for ultra-wide screen support
• Fix for audio levels not being applied properly after the update in certain cases
• Additional fix for sliders not working when dragged along
• Fix for RT shadows switching when looking through a portal on Valravn’s map
• Fix for stuttering some people experience with DX12, done by pregenerating a package of shaders for both RT On and Off presets.

Good to hear! And I strongly believe that Alex bringing it up in that DF Direct, as well as just in general in other videos has a lot to do with these things actually getting fixed! Also, this bodes well for Hellblade 2, as theoretically they wouldn't want to make the same mistake twice.

 

[Malo]

Isn't the whole problem with shader generation on PC that you have to generate them per GPU and driver version? Or is this doing shader generation on startup?

 

[Remij]

Isn't the whole problem with shader generation on PC that you have to generate them per GPU and driver version? Or is this doing shader generation on startup?

I haven't tested it yet, but I'm assuming they mean locally pre-generating on startup.

 

[Scott_Arm]

Isn't the whole problem with shader generation on PC that you have to generate them per GPU and driver version? Or is this doing shader generation on startup?

It's doing it on startup instead of during gameplay. The better way of doing it. Yah, the problem is you have to compile shaders per gpu per driver, so you can't ship them all with the game.

 

[Remij]

Ok well I just tested it, and there was no pre-compilation process for me... and the game still stutters like crazy in DX12.

 

[Remij]

Got another one here.. wow... Guerrilla Games adds DLSS and FSR to Horizon Zero Dawn on PC, and again improves shader compilation process!

https://twitter.com/i/web/status/1468521127358046208

Hello all,

We’re happy to announce we’ve just released Patch 1.11 for our PC players. Here’s what this patch contains:

Graphical Improvements

• Added Nvidia’s DLSS upscaling technology.
• Added AMD’s FidelityFX Super Resolution, replacing FidelityFX CAS.
  

UI Changes

• Adjusted settings screen to facilitate the addition of DLSS and FSR.
  • Render Scale option has been removed but same result can now be accomplished by adjusting setting Upscale Method to Simple and adjusting Upscale Quality.

Performance Improvements

• Improvement to the shader management system. This will result in a few noticeable differences:
  • There is no longer a shader pre-compilation step on startup. The game will always compile shaders during loading and in the background.
  • Stutters during gameplay that used to occur due to background shader compilation have now been significantly reduced.
  • Because shader compilation is still happening in the background you may notice the game having a higher CPU utilization while that is happening.
  • Loading screens will wait for the required shaders to be fully compiled. This may cause loading screens to take somewhat longer on certain systems.
  • On higher spec machines with faster CPUs the loading screens will typically be shorter, due to more efficient shader compilation that better leverages high-end CPUs.

Please ensure your game is up-to-date before heading back out into the wilds, and reach out to us if you’re still experiencing any issues. We appreciate all of your wonderful support and feedback; we wish you a fun-filled festive period!

- Guerrilla

 

[BRiT]

Weird that they remove the separate step and instead shift it to in the background process. Though it sounds like they made it more efficient and has loading screens waiting for the process to complete before proceeding on.

 

[DegustatoR]

It was a background process since patch 1.09 I think.
The difference is that they've removed the indicator of shaders compiling from the main menu.
Dunno if that's an issue as I never really saw much difference between waiting in main menu for this compilation to finish and just loading the game straight away.

 

[Remij]

Ok, so I've tested it out, and they indeed have removed the

It was a background process since patch 1.09 I think.
The difference is that they've removed the indicator of shaders compiling from the main menu.
Dunno if that's an issue as I never really saw much difference between waiting in main menu for this compilation to finish and just loading the game straight away.

Yea. Though to be honest, I wish they'd add that indicator back in. Keep it the same way it is now, but just add the indicator back to the main/pause menu so that people would know when the game has finished fully caching the shaders without having to use some program to monitor their CPU usage to let them know.

 

[DegustatoR]

I mean I can see this being an issue on slower CPUs where background compilation would either affect general performance or won't be able to finish in time when the shaders will actually be needed for a frame.
So some sort of process indication is generally a good idea.

 

[Lurkmass]

Shameless self-plug incoming and we have possibly more bad news on the way ...

https://twitter.com/i/web/status/1468723900725006336

 

[Remij]

^yay...

I'm so tired of this BS. FF7R just released today, and yep... sure as hell.. it's got stuttering issues all over the place from shader compilation...

Look at this... your first introduction to the world of FF7R..

You know, it's actually a nice metaphor.. as the cinematic intro video ends on a train engine screeching to a halt the UE4 render thread also screeches to a halt any time it has to wait for shaders to compile.


...like... who QA tests this stuff? How do you let it ship like that? I just don't get it. It's god damn depressing as hell to me. Then when Cloud jumps down from the train... another massive stutter...

This game is cinematic as hell... it's going to be like that the entire damn time.. ugh.

 

[Lurkmass]

I thought everyone knew that Unreal Engine compiles at draw time ? Pretty sure compilation stutters are intentionally by design in Unreal Engine for the sake of the artists or iteration times ...

Artists would prefer stutter over their editor freezing/locking up for potentially indeterminate amount of time ...

 

[Remij]

I thought everyone knew that Unreal Engine compiles at draw time ? Pretty sure compilation stutters are intentionally by design in Unreal Engine for the sake of the artists or iteration times ...

Artists would prefer stutter over their editor freezing/locking up for potentially indeterminate amount of time ...

...lmao

 

[Dictator]

^yay...

I'm so tired of this BS. FF7R just released today, and yep... sure as hell.. it's got stuttering issues all over the place from shader compilation...

Look at this... your first introduction to the world of FF7R..

You know, it's actually a nice metaphor.. as the cinematic intro video ends on a train engine screeching to a halt the UE4 render thread also screeches to a halt any time it has to wait for shaders to compile.


...like... who QA tests this stuff? How do you let it ship like that? I just don't get it. It's god damn depressing as hell to me. Then when Cloud jumps down from the train... another massive stutter...

This game is cinematic as hell... it's going to be like that the entire damn time.. ugh.

So this is probably one of the worst AAA PC ports in a long time. Impressive.

 

[Lurkmass]

So this is probably one of the worst AAA PC ports in a long time. Impressive.

Unreal Engine's inherent limitations aren't the fault of developers ...

 

[Remij]

Unreal Engine's inherent limitations aren't the fault of developers ...

Explain Doublefine who managed to fix Psychonauts 2's compilation issues then.. or The Coalition with Gears.. or Sony Bend with Days Gone...

This isnt an inherent engine issue... You CAN precompile shaders in Unreal Engine... developers just seemingly don't bother to. There are developers who care to put in the effort to properly optimize.. and those who don't.. or are too small and can't due to lack of resources.

SquareEnix is not a small developer... and a game like FF7R isn't a low budget game.. this is embarrassing to ship a game like this..

 

[Lurkmass]

Explain Doublefine who managed to fix Psychonauts 2's compilation issues then.. or The Coalition with Gears.. or Sony Bend with Days Gone...

This isnt an inherent engine issue... You CAN precompile shaders in Unreal Engine... developers just seemingly don't bother to. There are developers who care to put in the effort to properly optimize.. and those who don't.. or are too small and can't due to lack of resources.

SquareEnix is not a small developer... and a game like FF7R isn't a low budget game.. this is embarrassing to ship a game like this..

The default behaviour for Unreal Engine is asynchronous shader compilation with on-demand PSO streaming. Basically, every time the engine issues a draw command and there's a unique PSO you are guaranteed to get shader compilation related stutters ...

Developers could try to implement a workaround but why would Square do that in particular when their project isn't even complete yet ?