Shader Complilation on PC: About to become a bigger bottleneck?

Discussion in 'Rendering Technology and APIs' started by Flappy Pannus, Aug 6, 2020.

  1. Remij

    Newcomer

    Joined:
    May 3, 2008
    Messages:
    232
    Likes Received:
    385
    Exactly. That's why I brought up that we have to wait to download a game, and have to wait again to download it if we delete it to make room on our storage drives. It's a necessary evil, and we wait that time with no issues what so ever.

    Having a 10min process happen once the very first time you launch a game, and most likely never again..or very rarely, is absolutely nothing in the grand scheme of things. Now if your game is performing like junk and hitching all the time.. that's a far more detrimental to the experience.

    People just have to think of it like an extended install process. One phase where you download the game, and the other where you need to run it to finish "installing" it.
     
    digitalwanderer likes this.
  2. Flappy Pannus

    Newcomer

    Joined:
    Jul 4, 2016
    Messages:
    145
    Likes Received:
    172
    This comparison would be applicable if you had to download your games anew after every driver or US update.

    One of the biggest complaints about the Epic launcher btw is that it can't simply recognize currently existing installed games from a previous OS installation, if for example you took away Steam's ability to just recognize games that are already installed and do a 5-minute verification pass but required you to re-download them every time, even with the relative infrequency of OS reinstalls/updates you would never hear the end of the screaming on the forums.
     
    #42 Flappy Pannus, Aug 10, 2020
    Last edited: Aug 10, 2020
    BRiT likes this.
  3. Flappy Pannus

    Newcomer

    Joined:
    Jul 4, 2016
    Messages:
    145
    Likes Received:
    172
    If simply doing shader pre-compilation as a whole before the game launched was a panacea for game hitching on PC (as you seem to believe), I would probably be far more acceptable of it.

    But...it isn't? There are many reasons that a game can be stuttering on PC. H:ZD has a length shader compilation process - it still stutters like crazy on many configs atm. Detroit:Beyond Human pre-compiles its shaders in a long process as well, it can still stutter. I can get stuttering in Dishonored 2 due to texture paging that are not related to shader compilation, which it does on load frequently (even outside of driver updates - and again, as we're seeing with HZD, re-engaging shader compilation automatically on driver updates is the least-frequent implementation possible). Arkham Knight's stuttering is not due to shader compilation either.

    We agree that stuttering in-game is far more detrimental to the experience than a lengthy pre-compiling stage, obviously as a % of gametime, even if I play the game interspersed between driver updates and have to see that compiling screen several times before I complete it, it comes out to a relatively small impact overall relative to the time I wait in game loading other elements. I just see little evidence that introducing the pre-compiling step is some sort of stuttering cure-all.

    More to the point of my first post though, is that we're soon going to have consoles that don't have this problem, but also will not have the load time issues that they do now inside of the game. As said, while the lengthy pre-compile process is annoying, atm at least compared to consoles, the overall experience of waiting for game assets to load on PC is far superior in terms of returning to the action after a death, fast traveling, etc - game like Doom 2016 for example is far more enjoyable on hard difficulties on my PC over my PS4 in part because when I die I'm back in the action in seconds.

    But very soon, that will no longer be a distinction, in fact it will likely go the other way, especially with consoles implementing state resume. It's one thing to no longer have a large advantage for a more expensive platform, it's another to actually have a detriment. I wonder if a gamer used to bouncing between games on their PS5/Xbox Series X in seconds loads up a game in PC, and after waiting for that game to actually load (as there is no save state functionality), then has to stare at a 'recompiling shaders' screen for 15-20 mins. I can imagine them thinking 'wtf is this shit?'. I think that's a problem.
     
    BRiT likes this.
  4. MfA

    MfA
    Legend

    Joined:
    Feb 6, 2002
    Messages:
    7,314
    Likes Received:
    670
    With 8+ cores and proper storage QoS why should it cause stuttering to begin with? Assuming it pre-compiles all the shaders necessary for the current level (or local zones for open world) and then does the rest in the background.

    It seems more a failure of the code and/or the OS, than background compilation in principle.
     
  5. Lurkmass

    Regular Newcomer

    Joined:
    Mar 3, 2020
    Messages:
    306
    Likes Received:
    345
    Somewhat unrelated but there is a way to improve the situation which is to expose some API extension like having more dynamic rendering states available in the drivers. This can help reduce the number of PSOs needed because more of the graphics state can be set outside of the pipeline and as a result it cuts the length of the shader compilation time as well but the added irony is that this pretty much goes against the stateless design of the new gfx APIs which was what led to the downfall of the old gfx APIs ...

    Nevertheless, I trust that the HW vendors know what they're doing with that extension since they can at least guarantee that changing those rendering states won't cause shader recompilations behind our backs ...
     
  6. Ethatron

    Regular Subscriber

    Joined:
    Jan 24, 2010
    Messages:
    921
    Likes Received:
    356
    To me it's a failure of the market principles. As this mechanism becomes more complex, it should be distributed to a player that focuses on only solving these aspects, and provide a stable, useful middle-ware. Besides the big engine players adopting other middle-wares, there is a broken development culture going on, preventing these adoptions. "It cost", "We can do it", "We can do it better", "We can't deal with the bugs", "Asking for new features and corrections takes aaages", etc. blabla. Then some IHVs think they can gain a competitive advantage to give out utility libs/code, but it only works on their hardware, or in their eco-system, is even more unconstructive noise in an already exhausting dev cycle.

    In terms of collaborative development of rock solid foundations/solutions for complex multi-platform and multi-hardware management issues and programming patterns, game development is not coping well in the PC space. Some console vendors are very good, they give you huge amounts of utility/code, which they improve and which is the (potential) baseline for almost every game on their platform. Others not so, they make it their philosophy to give freedom to everyone to implement it in their own needed optimal way, feeding into the broken development culture dead-loop above, and apparently consciously taking into account that QA for their platforms is all over the place.

    Many challenges are larger than one game developer, and sadly, over short or long term, the big engine players will be the only ones able to provide proven Quality of Service for their lower level sub-systems behaviour, and all the others will have to "cope". It's sad, as there is very little counter-movement towards constructive collaboration, that I can see. One big good example is DXC and the SPIR-V tools, it blooms and it's great (from a culture perspective).
     
  7. Alessio1989

    Regular Newcomer

    Joined:
    Jun 6, 2015
    Messages:
    605
    Likes Received:
    320
    What could be going wrong? oh wait anything XD

    The real question is: what kinda shader sources provides those games, IL/bytecode or just source/assembly?
     
  8. Per Lindstrom

    Newcomer Subscriber

    Joined:
    Oct 16, 2018
    Messages:
    51
    Likes Received:
    51
    Location:
    Sweden
    Interesting topic.
    One way to implement it, could be a right-click on game-icon, and then select compile shaders in background.
     
    #48 Per Lindstrom, Aug 22, 2020
    Last edited: Aug 22, 2020
    milk likes this.
  9. Per Lindstrom

    Newcomer Subscriber

    Joined:
    Oct 16, 2018
    Messages:
    51
    Likes Received:
    51
    Location:
    Sweden
    Lack of memory?
    DX12 need more memory then DX11, i have been told.
    Maybe this is one reason?
     
    BRiT likes this.
  10. MfA

    MfA
    Legend

    Joined:
    Feb 6, 2002
    Messages:
    7,314
    Likes Received:
    670
    If a shader compiler needs lots of memory something is very wrong with the implementation too.
     
  11. Per Lindstrom

    Newcomer Subscriber

    Joined:
    Oct 16, 2018
    Messages:
    51
    Likes Received:
    51
    Location:
    Sweden
    It may be so, that's the case for Horizon Zero Dawn.
    12 GB is extreme, so yes, something is very wrong.
     
    BRiT and TheAlSpark like this.
  12. MfA

    MfA
    Legend

    Joined:
    Feb 6, 2002
    Messages:
    7,314
    Likes Received:
    670
    No excuse really, this isn't compiling desktop spaghetti code. Someone needs to fix their shit.

    I don't see how it would be impossible to use resource limits to keep the background compilation at a level where it should not impact the consistent performance of the game. It might consistently lower it a bit, but it shouldn't jump up and down.
     
  13. Flappy Pannus

    Newcomer

    Joined:
    Jul 4, 2016
    Messages:
    145
    Likes Received:
    172
    That's actually an interesting idea, perhaps games would report to Windows when they detect shader compilation is necessary and you have an option in Windows settings you can enable to 'compile game shaders during idle', basically a low-priority process that would fire up when necessary. You'd definitely have to be careful to make it get the hell out the way when you actually need those CPU cycles though, it's one thing to get shader stuttering in the game it's actually compiling for, it's another to induce stuttering for compiling shaders for a game you're not even playing. :)
     
    Remij and Per Lindstrom like this.
  14. Remij

    Newcomer

    Joined:
    May 3, 2008
    Messages:
    232
    Likes Received:
    385
    That's what I mean. Just give me the option to do it, and I'll absolutely take advantage of that.

    What's 10 or 15min of an "install process" added on to hours long download times for some people?

    It's when you click "play" and you expect to just be able to get into the game and THEN have to go through another process is when it starts to annoy people.

    Making it a part of the Steam Download/Install process would likely solve most of the arguments with it.
     
    milk, CeeGee and digitalwanderer like this.
  15. Flappy Pannus

    Newcomer

    Joined:
    Jul 4, 2016
    Messages:
    145
    Likes Received:
    172
    Doing that would help a bit, but the big issue with shader compilation is it needs to be done with every driver update, and in some cases, game patches. If it just needed to be run once during an install I likely wouldn't have ever thought of even creating this thread.
     
    digitalwanderer likes this.
  16. Remij

    Newcomer

    Joined:
    May 3, 2008
    Messages:
    232
    Likes Received:
    385
    Well once again, with game patches the process could be done through the Steam process.. and I still honestly don't see people changing drivers often through a single playthrough of a game. Once a month you have to deal with a 5-10min "optimization"... not a big deal at all, and a worthy sacrifice for smoother gameplay. Again, that's just me.

    So if you essentially hide 2/3rds of the times when this process would need to happen, then you're still better off than you were before.
     
    milk likes this.
  17. Per Lindstrom

    Newcomer Subscriber

    Joined:
    Oct 16, 2018
    Messages:
    51
    Likes Received:
    51
    Location:
    Sweden
    One strategy can be to compiling a slow version of the shaders in short time, and then pop up a question window.
    Code:
    Your Shaders, for this game, are not optimized, this takes rather long time.
    Optimized Shaders increase performance.
    
    Do you want to?
    
    * Play now, you will be asked if you want to optimize Shaders, after finished playing.
    
    * Play now, and optimize Shaders while you play, this may decrease game performance, during optimization.
    
    * Optimize Shaders before you play, you may use your computer as usual, during optimization.
    
        (OK)   (Cancel)
     
    #57 Per Lindstrom, Aug 25, 2020
    Last edited: Aug 25, 2020
  18. Davros

    Legend

    Joined:
    Jun 7, 2004
    Messages:
    16,751
    Likes Received:
    4,049
    question : with hundreds of games installed why do i have no memory of these incredibly long delays caused by shader compilation ?
     
  19. BRiT

    BRiT (>• •)>⌐■-■ (⌐■-■)
    Moderator Legend Alpha

    Joined:
    Feb 7, 2002
    Messages:
    18,795
    Likes Received:
    21,097
    You've been buried in your TB of games from the 90s? Not running DX 12 enabled software and hardware?
     
  20. DegustatoR

    Veteran

    Joined:
    Mar 12, 2002
    Messages:
    2,206
    Likes Received:
    1,601
    Location:
    msk.ru/spb.ru
    Because most games manage to do it properly.
     
Loading...

Share This Page

  • About Us

    Beyond3D has been around for over a decade and prides itself on being the best place on the web for in-depth, technically-driven discussion and analysis of 3D graphics hardware. If you love pixels and transistors, you've come to the right place!

    Beyond3D is proudly published by GPU Tools Ltd.
Loading...