strategy for automatic shader compilation caching (with change detection support)

[gjaegy]

Hi,

I am trying to reduce the load time of my game.

Right now, I compile all the shader at runtime, when the game is started, which takes quite some time.

I use D3DX10CompileFromMemory to compile my shaders.

I am thinking about writing the content of the blob returned by this function on disk, the first time the shader is being compiled, and re-use it next time the game is started.

This is very straightforward.

However, what I also would like to achieve, is to automatically detect any change in the shader source, and re-compile the shader if any change occured.

So, my initial idea was to save two files, the source code version used for compilation, and the resulting compiled shader blob.
At runtime, I would load the shader source file, and compare it with the source version used for compilation. If there is no difference, i can use the pre-compiled blob, otherwise I compile the shader again.

However, this approach gets a bit more complicated when considering "#include" directives, along with the usage of a "ID3D10Include" object.

The only way I see would be to manually resolve all the #include" directives, in order to get a "#include"-free HLSL source code I could use to compare the current version with the pre-compiled cache version.

Is there any other option I haven't seen ? How do you guys handle this problem ?

 

[Davros]

why do you need to do this ?
you know when youve changed the shader, if this is a time saving thing during developement just precompile them
If its a time saving thing after release, its not needed end users dont change shaders

So, my initial idea was to save two files, the source code version used for compilation, and the resulting compiled shader blob.
At runtime, I would load the shader source file, and compare it with the source version used for compilation.

This sentence doesnt quite make sense
you only have 2 files yet the above quote mentions 3
The source code version used for compilation, the resulting compiled shader blob and the shader source file

 

[gjaegy]

Thanks for your answer, Davros.

Why do I need that: basically, I want the system to work in both dev and production environment. After release, an upgrade/patch might be applied, so the shaders can change too.

I suppose I could simply delete the cache manually each time I modify a shader (development) or when an upgrade is being applied (release). Now, this is something that require a manual operation, which I would like to avoid (error prone / confusing / time consuming).

Regarding the number of files, I realize I am not very clear

So basically, what we have at start is a HLSL source file. Lets call it "source.hlsl".

At the very first time, this file gets compiled (as the compiled shader cache is empty). In the cache folder, we are going to write two files:
- the reference HLSL file used for the compilation => "source.ref" (which is simply a copy of "source.hlsl" for now)
- the resulting byte code=> "source.bin"

Next time the game is started, the engine loads "source.hlsl" and compare it to "source.ref". If both files are the same, the engine assume it can use "source.bin" and avoid the compilation process.

Now, imagine that for whatever reason, "source.hlsl" change. The game is started, "source.hlsl" and "source.ref" are compared, but this time a difference is found. The engine assume "source.bin" is outdated and "source.hlsl" is being compiled again ("source.ref" and "source.bin" get updated then).

Does that make sense ?

 

[AlexV]

Something like a hash map should work I think. Index into it by key, compare hashes to see if modified...from a bird's eye view it appears as if it'd be a decent solution.

 

[Davros]

Thanks for taking the time gjaegy, thats a great answer.
but wouldnt you be better shipping pre compiled shaders (what are the advantages of run time compiling)
BF2 seems to do something similar, but it seems to recompile the shaders every time I update gfx card drivers and its annyoing

 

[3dcgi]

Thanks for taking the time gjaegy, thats a great answer.
but wouldnt you be better shipping pre compiled shaders (what are the advantages of run time compiling)
BF2 seems to do something similar, but it seems to recompile the shaders every time I update gfx card drivers and its annyoing

Compilers are continuously being improved so if you want your customers to receive these benefits you recompile when there's a new driver.

 

[MJP]

Compilers are continuously being improved so if you want your customers to receive these benefits you recompile when there's a new driver.

The HLSL compiler isn't part of the driver, it's part of the D3DCompiler DLL that's shipped with the SDK (and optionally shipped out to end users via the redist packages). So unless you re-compile your app linked to a new version of that DLL you're not going to use a new version of the compiler. There's the JIT compiler in the driver that will compile your shader bytecode into hardware-specific microcode, but that's always going to happen at runtime no matter how you compile HLSL shaders. So as far as I'm concerned there's no reason to compile shaders at runtime unless your shader code is going to somehow change after being deployed to the user's machine.

In our engine at work (and most other engines that I'm familiar with) shaders are just one of many types of assets that need to be pre-built/compiled before runtime and that need to track dependencies. So usually they make use of a much larger/more complex pipeline that handles much more than just shaders. That said, you could still have a runtime dependency check/compilation step for development purposes that checks for modified files and hot-loads them in. For shaders I would definitely recommend using some kind of hash rather than hanging onto the source code.

 

[Psycho]

Generally it's the step after the precompile (ie the actual compilation by the driver) that takes the time. So you won't win much by caching your shaders anyway..

 

[3dcgi]

I figured the original post was talking about the hardware specific compiler since Davros said BF2 recompiles with each driver update.

 

[reptile]

gjaegy, what you describe is exactly what im doing in our engine. In my case, i'm storing the hash of the original source code, and some additional parameters (like entry point, variation number) in the file name of the shader binary. Then at runtime (or more importantly, in the editor) i generate the hash of the actual source and try to load the binary. If it doesn't exist, it means the source/entry point/variation has changed and needs to be (re)compiled, so basically i'm using the file system (or the package system in our case) to perform a hash lookup. I realise it's not as efficient as just storing the hash for everything, but unless you have tens of thousands of shaders, it shouldn't make a difference.

 

[Ethatron]

Just use D3DXPreprocessShader with an include-manager to call the pre-processor. You get a similar result as "gcc -e", pure independent language-code. As it's the pre-processor only it doesn't matter which profile you use, the hlsl syntax isn't even looked at.
That result then you can save, checksum or compile with D3DXCompileShader.

 

[gjaegy]

Thanks guys for the answers !

I am now done with this, and am pretty happy with the result.

First of all, I resolve all includes manually. This results in a big HLSL source code text with no #include directives anymore. (I wasn't aware of D3DXPreprocessShader not a big deal though).

I then check the shader cache (which contains a pair of "hlsl source code" / "compiled byte code" files for each original shader). In the cache, the shaders are referenced by HLSL file name / entry point / CRC of preprocessor definition values string (to handle variations).

If an entry already exists in the cache, I compare the current source code text with the "hlsl source code" file. If no change could be found between the two strings, I assume I can safely load the existing "compiled byte code" file and use it.

Otherwise the shader gets compiled, and the result saved on disk ("hlsl source code" / "compiled byte code" files).

It seems to work pretty well.

I guess I could also generate a hash of the HLSL source text, instead of comparing the whole string. However, I am a bit worried of hash collision cases, where changes would not get detected if the same hash code is produced. In the other hands the string compare doesn't seem to be that expensive, considering it is done once per shader at startup...

I am pretty happy with the result, as there is no need additional task required when a new version of the software is being released. Simply pack the shaders source files along with the other data, the system will then automatically compile the updated shaders the first time the updated software is being launched by the customer.

 

[mikiex]

A possible advantage of not precompiling is that users can easy rewrite your shaders to improve them or get advantages in multiplayer etc.