Shader Compilation on PC: About to become a bigger bottleneck?

[Remij]

This is like a cynical hellscape opinion about software, and if that's the opinion developers take towards releasing games we're all fucked. The idea that a person charging money for a product shouldn't care about their user experience because no one else does is just absolutely shitty. If people buy shitty games, they should refund them. Refund policies are important. Literally every single person that sells any product, whether it's software or knives or paper, should care about the quality of the product they're putting out into the world.

Yep...It's just complete BS. I don't know what developers REALLY think or do behind the scenes... and honestly I don't really care.. we all hate our jobs sometimes.. All I'm asking for is that games perform as they should on a BASIC LEVEL. Hitching.. where your game is ACTUALLY FREEZING is NOT acceptable! As a consumer all I can do is vote with my money, and basically yell about it on forums.. and I hate that the only way to invoke any kind of change, is to make a big stink where it catches wind and goes "viral".. That's why I feel like people like Alex from DF are so important. Digital Foundry pointing out egregious issues in PC games is the most likely the best chance we have for these issues to gain mass traction and become visible to many of these developers. I REALLY appreciate the work they do, and man I hope when Alex makes his video regarding this issue with PC games he really digs in on it.. with no mercy shown.

It's really disheartening for me though. I feel like there are solutions to this issue, but only a small amount of developers actually care... which is why when I find a developer who actually listens and encourages feedback to improve their products.. and indeed actually fixes their product... I try to give them all the praise I can and spread the word. Double Fine actually going back and doing the work to fix the issue is the latest example for me, and that's an UE4 game to boot..

Sooner or later it's going to get bad enough to the point where they can't ignore it anymore. I've almost NEVER refunded a game in my life.. no matter how bad it was. I can look past small issues and certain bugs with hopes that a future patch will correct them... but people like Lurkmass here acting like this is actually a small issue just completely blows my mind. This is absolutely the single biggest issue facing PC games today. You can have all the "average FPS" in the world... and it doesn't matter one damn bit if it's constantly hitching and stuttering. That is NOT how games should function. This is supposedly the best engine in the world... how the hell are we dealing with this stuff with our massive 8/12/16 core CPUs with 16/24/32 threads?? Why the HELL aren't developers properly testing their games and finding these issues and correcting them before they are released? The smaller developers I can somewhat understand... but with bigger AA and AAA development studios.. it's unforgivable.

I've said this before... I'll say it again.. I really don't care what this industry has to do to fix this issue... but they HAVE to do something or I'm simply done with it. It's not my job to understand how to build a car.. I simply drive the damn thing.. and I expect it to perform a certain way. Same thing applies here with games. MS, AMD, Nvidia, Intel, engine developers, game developers... you guys need to figure out some god damn solution and stop pointing the finger at anyone else... games have to stop releasing in this kind of state.. period.

 

[MfA]

Why don't AMD or NVIDIA make an unreal plugin together with a best practices guide which lets the game just download a cache for supported hardware and recent drivers? Ohh they have to compile and store 1000s of variations ... so what? CPU time and storage are cheap.

 

[Lurkmass]

This is like a cynical hellscape opinion about software, and if that's the opinion developers take towards releasing games we're all fucked. The idea that a person charging money for a product shouldn't care about their user experience because no one else does is just absolutely shitty. If people buy shitty games, they should refund them. Refund policies are important. Literally every single person that sells any product, whether it's software or knives or paper, should care about the quality of the product they're putting out into the world.

Commercial engines just like many other external dependencies are double edged swords. They can give you more productivity but they take away your control over the technology. It's really not that hard to understand why programmers are hesitant to touch code created from other programmers ...

The compilation issue is largely stuck in limbo because the problem is out of control for most developers and in Square's case where Epic Games is their official partner who is more than capable of providing technical support they simply don't care since the end users aren't their customers and they will gladly just greenlight the release! All Epic Games want is growth for their own digital store so blocking a release for a technical issue that hasn't harmed them in the past is against their own self interests ...

Consumers could try to take a principled stand by blacklisting any products that uses the Unreal Engine until developers raise this issue to Epic Games for them to fix it but I have very few hopes of this ever happening since they weren't interested in sending the right message in the past ...

 

[DavidGraham]

I recently played the game A Plague Tale, wondeeful game with graphics very similar to Unreal Engine games, the difference is that the game renders wide open areas smoothly and without a single hitch, to my amazement of course, as I thought the game is using Unreal Engine, and I couldn't believe a UE game is running this smoothly without a single stutter! But then I discovered later that it's a custom made engine.

So what's the deal with shader compilation anyway? They are compiled by the CPU, so why do they have to be compiled so specifically to begin with?

As I understand it, it's a form of real time optimization (JIT or Just In Time) to convert some generic code into machine code that is specifically tuned to every GPU architecture, so in essence it's trying to eeke out every single GPU power from the hardware, to increase fps.

But then GPU architectures are not that many, for a typical DX12 game (DX12 titles suffer the most from this issue), you only have these architectures to support: GCN, Polaris, Vega, RDNA1 and RDNA2 from AMD. NVIDIA has Kepler, Maxwell, Pascal, Turing and Ampere. So really not that many! What's stopping developers from precompiling the shaders for these ten architectures, while leaving the real time compiling exclusively for future archs only?

And while we are at it, Why not give the user the option of running the generic code for stutter free experience and slightly reduced fps?

 

[Scott_Arm]

Commercial engines just like many other external dependencies are double edged swords. They can give you more productivity but they take away your control over the technology. It's really not that hard to understand why programmers are hesitant to touch code created from other programmers ...

The compilation issue is largely stuck in limbo because the problem is out of control for most developers and in Square's case where Epic Games is their official partner who is more than capable of providing technical support they simply don't care since the end users aren't their customers and they will gladly just greenlight the release! All Epic Games want is growth for their own digital store so blocking a release for a technical issue that hasn't harmed them in the past is against their own self interests ...

Consumers could try to take a principled stand by blacklisting any products that uses the Unreal Engine until developers raise this issue to Epic Games for them to fix it but I have very few hopes of this ever happening since they weren't interested in sending the right message in the past ...

I get the part where Epic shouldn't be off the hook either. Their customers are game developers, and developers should be pushing them to solve the issue in engine. They're paying for the engine and it should have solutions for common issues. Still doesn't change the fact that Square Enix, or any other company is ultimately responsible for releasing quality products. If their game ships with stuttering and freezing, it was their choice to release it that way and they were the ones that charged money for it. They own 100% of what they put out the door to their customers. If they blame Epic's engine, ultimately I don't care. I'd refund the game if it had those problems.

 

[DavidGraham]

One way to implement it, could be a right-click on game-icon, and then select compile shaders in background.

Geforce Experience tracks every game you ever play on your system, it has CPU/GPU specifc optimization profiles for thousands of combinations and it can manipulate the settings for thousands of games without ever launching them to optimize the performance for the user with one click, I don't see why it shouldn't automate the process of shader cimpilation for dozens of games in one click too.

 

[Remij]

Geforce Experience tracks every game you ever play on your system, it has CPU/GPU specifc optimization profiles for thousands of combinations and it can manipulate the settings for thousands of games without ever launching them to optimize the performance for the user with one click, I don't see why it shouldn't automate the process of shader cimpilation for dozens of games in one click too.

All we REALLY need, is for developers to implement the option for us to do it ourselves locally first and foremost. Once you get that way of thinking ingrained, then you can look at ways to speed up that process using outside means.

I mean, these companies like Nvidia and AMD have massive servers with insane amounts of computational power that could crunch through this stuff insanely fast. Nvidia/AMD would really do well for themselves to create an API which would allow developers to easily integrate server side compilation into their game, so then the user runs this process (or doesn't their choice) and the server kicks in and blasts through the compilation process way quicker than you could ever hope to. That way, instead of waiting 5-10 minutes, it takes like 30 seconds to 1 minute.

They of course could make it do this automatically.. The game detects you're connected to the net, and automatically engages the server to assist with the process.

I mean, these hitches and stutters just make Nvidia's and AMD's overpriced PC hardware look terrible. It's absolutely in their interests to solve this problem. They apparently need a wakeup call.. and I just don't know anymore... they're selling more GPUs than ever at prices higher than ever. It's maddening... ffs.

 

[Remij]

PC Gamer actually just put up an article about how bad the PC port of FF7R is. Which I'm glad for. It really does need more attention so SquareEnix will maybe do something about it.

I mean, ok this video is obviously another issue altogether than shader compilation.. but it just goes to show the quality of the port. The shader stutters happen all the time during cutscenes and transitions..

But I mean look at this....

Are you SERIOUSLY telling me that SquareEnix tested this game... and if they did.. that they thought this was acceptable?? That's not Epic's fault... that's SquareEnix's fault. There's NO excuse for this. And this damn game costs $93 CAD....

That's why I say this can't keep going on... we really need to start sticking it to these companies that try this crap.

 

[Scott_Arm]

PC Gamer actually just put up an article about how bad the PC port of FF7R is. Which I'm glad for. It really does need more attention so SquareEnix will maybe do something about it.

I mean, ok this video is obviously another issue altogether than shader compilation.. but it just goes to show the quality of the port. The shader stutters happen all the time during cutscenes and transitions..

But I mean look at this....

Are you SERIOUSLY telling me that SquareEnix tested this game... and if they did.. that they thought this was acceptable?? That's not Epic's fault... that's SquareEnix's fault. There's NO excuse for this. And this damn game costs $93 CAD....

That's why I say this can't keep going on... we really need to start sticking it to these companies that try this crap.

5900x and RTX3080 and it runs like that. Brutal. That's definitely a refund title.

 

[Flappy Pannus]

Good thread by @Dictator on FF7R's stutter and the less than optimal method many PC benchmarking sites use to evaluate framerate on PC games. Definitely with him especially on the point of 1% lows being inadequate to evaluate frametime consistency.

https://twitter.com/x/status/1472485360844300291

 

[BRiT]

Unrolled Twitter thread linked @ https://threadreaderapp.com/thread/1472485360844300291.html

 

[jlippo]

I doubt developers of the port had any choice with the release day.
Really looks like a small team was set to the task and told it launches in couple of weeks.

 

[Rootax]

I doubt developers of the port had any choice with the release day.
Really looks like a small team was set to the task and told it launches in couple of weeks.

Agree, but it's the case with most project, devs don't chose the release date. Most likely they didn't put the best people to the task either...

 

[Remij]

Again, this doesn't seem to be shader compilation related, as it continually happens in the same spots all the time... but it just goes to show how terrible Unreal Engine is when loading in new assets.. Approach the boundary of a level connecting something else and you get hitches..

Look at this...

Who wants to play that? Definitely not me... Imagine spending $3000+ on a powerful PC... to experience the game like that..

Tales of Arise in my experience had ZERO compilation stuttering, the game was always silky smooth for me, until you approached the border of a level, such as a door or something, then you'd get a hitch as you approach.

It's just depressing, because the engine is obviously awesome at so many things... but you have these compilation stutters and hitches when loading in new areas... it just completely destroys the immersion. The only Unreal Engine games which in my experience don't do this, are the ones which are heavily customized versions where the developer has tailored the engine to suit their needs.

So the reason why this pisses me off, is that it's obviously fixable... but smaller developers might not have the resources to fix it.. and thus we have tons of games which release with these issues. For a company like SquareEnix though? There's absolutely NO excuse. They just don't give a damn.

I blame developers first and foremost because it's THEIR product they are selling and playtesting and quality assurance should bring these issues to light and they should be fixed before release... but at the same time Epic should do something to fix this default behavior so this doesn't happen in the first place.

 

[Rootax]

Maybe it doesn't happen in the first place but the devs don't have time or skills to adjust the behavior for their games ? What I mean is, you will never have a perfect behavior with a multiplateforme / multipurpose engine imo. Same with unity. You'll have to dig deep if you want smooth performances in most situations.

 

[MfA]

The porting companies deliver what's requested. If the publisher doesn't care if the deliverable hitches they aren't going to waste time on it.

Those kinds of companies will port a ton of games, if the publishers ask for it they'll get the skills. They can focus on those kinds of problems far easier than primary devs.

PS. Google tells me object creation is on the single threaded game thread with lots of manual settings in the engine to spread that out across frames for streaming, debug objects will be heavier, could make the difference.

 

[Jawed]

Amongst other things, using a Nexus mod to turn off the game's dynamic resolution:

5950X+3090
Lifestream Reshade Mod +
No Dynamic Scaling Mod +
Tifa Purple (Refined) dress at all times Mod:

5800X + 6900XT
Lifestream Reshade Mod +
No Dynamic Scaling Mod:

Both systems have two "meaningful" stutters each during gameplay. They do look like it's a particular location in both cases, but I don't have the game and can't be bothered to watch the videos super-closely to compare.

Anyway, I think it's time to forget average framerates.

20 years ago we used to see "maximum fps" on graphs - nobody thinks about those numbers these days.

1% low and 0.1% low derived from a short run through the toughest part of the game are the way to go. 1% low really isn't that strenuous a measure. If we could only look at one number in a performance comparison then 0.1% low should be it.

 

[Lurkmass]

I blame developers first and foremost because it's THEIR product they are selling and playtesting and quality assurance should bring these issues to light and they should be fixed before release... but at the same time Epic should do something to fix this default behavior so this doesn't happen in the first place.

You can build a PSO cache in Unreal Engine but even Epic Games doesn't think it's realistic to achieve 100% coverage in many instances and there are other limitations too ...

https://twitter.com/x/status/1428387133677522949

This feature doesn't work with a popular VFX plugin or PSOs with ray tracing so no matter how hard many developers try to follow best practices including making super levels involving all assets, PC is forever a cursed platform in a lot of cases. The only way to permanently fix the problem for good is if Microsoft decides to make Xbox D3D available on PC so that that at least on one vendor with their latest architecture can reuse the same precompiled binaries on consoles. That won't ever happen since it will change the political balance between Microsoft and the HW vendors since it'll cause grief towards other HW vendors to see that one of them will have a consistent performance advantage. Having an "identical experience" between PC and consoles isn't technologically possible ...

 

[pjbliverpool]

Again, this doesn't seem to be shader compilation related, as it continually happens in the same spots all the time... but it just goes to show how terrible Unreal Engine is when loading in new assets.. Approach the boundary of a level connecting something else and you get hitches..

Look at this...

Who wants to play that? Definitely not me... Imagine spending $3000+ on a powerful PC... to experience the game like that..

Tales of Arise in my experience had ZERO compilation stuttering, the game was always silky smooth for me, until you approached the border of a level, such as a door or something, then you'd get a hitch as you approach.

It's just depressing, because the engine is obviously awesome at so many things... but you have these compilation stutters and hitches when loading in new areas... it just completely destroys the immersion. The only Unreal Engine games which in my experience don't do this, are the ones which are heavily customized versions where the developer has tailored the engine to suit their needs.

So the reason why this pisses me off, is that it's obviously fixable... but smaller developers might not have the resources to fix it.. and thus we have tons of games which release with these issues. For a company like SquareEnix though? There's absolutely NO excuse. They just don't give a damn.

I blame developers first and foremost because it's THEIR product they are selling and playtesting and quality assurance should bring these issues to light and they should be fixed before release... but at the same time Epic should do something to fix this default behavior so this doesn't happen in the first place.

Interesting that RivaTuner can sort out the frame pacing issues. It's a shame the developers weren't able to implement something so robust in their own code. I might start using Rivatuner on my games moving forwards to analyse frame time/pacing in this way in case there's any smoothness I've leaving on the table. Ah the joys of PC gaming!

 

[pjbliverpool]

You can build a PSO cache in Unreal Engine but even Epic Games doesn't think it's realistic to achieve 100% coverage in many instances and there are other limitations too ...

https://twitter.com/x/status/1428387133677522949

This feature doesn't work with a popular VFX plugin or PSOs with ray tracing so no matter how hard many developers try to follow best practices including making super levels involving all assets, PC is forever a cursed platform in a lot of cases. The only way to permanently fix the problem for good is if Microsoft decides to make Xbox D3D available on PC so that that at least on one vendor with their latest architecture can reuse the same precompiled binaries on consoles. That won't ever happen since it will change the political balance between Microsoft and the HW vendors since it'll cause grief towards other HW vendors to see that one of them will have a consistent performance advantage. Having an "identical experience" between PC and consoles isn't technologically possible ...

So is it not possible to pre-compile shaders for a game at load time in UE4 even if the developer wants to? i.e. UE4 forces this to be done in real time during gameplay?

And does anyone know how UE5 behaves in this regard?