Shader debug should turn off optimizations

[zgxnet]

Release Type: GitHub source current branch

Version: Latest commit at time of report

Platform(s): Windows

Describe the bug
In shader debug mode (setting by Game.EffectSystem.SetCompilationMode(Stride.CompilationMode.Debug), the compiled shader still has optimizations on.

To Reproduce
Steps to reproduce the behavior:

1. In program start, set shader debug mode using game.EffectSystem.SetCompilationMode(Stride.CompilationMode.Debug).
2. Launch using RenderDoc, and debug shader.
3. It is obvious that the compiled shader is hard to debug as as the optimizations are not turned off.

Expected behavior
The optimizations should be turned off in shader debug mode.

Screenshots
Not applicable.

Log and callstacks
Not applicable.

Additional context
The reason is clear. The main compiler function is Stride.Shaders.Compiler.Direct3D.ShaderCompiler.Compile. In the debug mode, the optimLevel is set to 0, and it calls SharpDX.D3DCompiler.ShaderBytecode.Compile, with flags ShaderFlags.None|ShaderFlags.OptimizationLevel0. In SharpDX, OptimizationLevel0 still has optimizations, to turn off optimization, it should be ShaderFlags.SkipOptimization. So a quick fix can be just a line of code:
Original code in Stride.Shaders.Compiler.Direct3D.ShaderCompiler.Compile:

switch (optimLevel)
{
    case 0:
        shaderFlags |= ShaderFlags.OptimizationLevel0;
        break;
......
}

fixed:

switch (optimLevel)
{
    case 0:
        shaderFlags |=  isDebug ? ShaderFlags.SkipOptimization : ShaderFlags.OptimizationLevel0;
        break;
......
}

[Ethereal77]

In my PR in which I have converted Stride to use Silk.NET instead of SharpDX, I've tried this. But I had to revert it immediately.

In principle, it's obvious that debug mode should not be optimized, but some tests in the test suite are limited explicitly to GraphicsProfile.Level_9_3 or below (for OpenGL, and for old Direct3D 9). In that profile, shaders are compiled in FeatureLevel_9_1, and that makes the tests fail if you don't optimize them, because some shaders in Stride are massive and end up using too many instruction slots, ALU slots, etc.

[zgxnet]

I believe that allowing good debuggability and supporting lower version profiles are not contradictory. We can allow optimLevel=-1, in which case we set SkipOptimization. Of course, a cleaner approach would be to map optimLevel 0-4 in Stride to SkipOptimization and OptimizationLevel0-3 in D3D respectively. This way we can satisfy all requirements while supporting lower versions and maintaining good debuggability.

[Ethereal77]

I believe that allowing good debuggability and supporting lower version profiles are not contradictory.

Of course, I agree with that.

We can allow optimLevel=-1, in which case we set SkipOptimization. Of course, a cleaner approach would be to map optimLevel 0-4 in Stride to SkipOptimization and OptimizationLevel0-3 in D3D respectively.

Note that this is not a D3D-specific thing or a OpenGL-specific thing. It is a compatibility problem, as GraphicsProfile_9_x represents support for very old GPUs.

Also, I already tried what you are suggesting. Look here. You can see I have the constant defined at the top of the class. But when you compile shaders for FeatureLevel 9.x, the resulting bytecode has some severe limitations, as old GPUs in the Direct3D 9 / OpenGL 1.2 era could only fit a few hundred bytes of shader bytecode at most, and with pretty strict limits on ALU instructions used, sampling instructions used, etc.

The only way we can lift this limitation and apply the D3DCOMPILE_SKIP_OPTIMIZATION flag is when we phase out anything below FeatureLevel 10. If we apply it now, many of the tests in Stride.Graphics.Tests, Stride.Graphics.Tests.10_0, Stride.Graphics.Tests.11_0, and many of the built-in complex shaders of Stride itself start to fail compilation.

[zgxnet]

I think you may have misunderstood my suggestion. Let me clarify a few points:

1. Conditional Enablement, Not Global Enablement

My suggestion is to enable SkipOptimization only for FeatureLevel 10.0 and above profiles, not globally for all profiles. The compilation failures you mentioned should only occur under GraphicsProfile_9_x. If different GraphicsProfiles are properly distinguished in the implementation, DirectX 10+ users won't be affected by DirectX 9 bytecode limitations at all.

2. Implementation Approach Issue

If your tests for Stride.Graphics.Tests.10_0 and Stride.Graphics.Tests.11_0 also failed after your attempt, this suggests there might be an issue with the implementation approach rather than the solution itself being infeasible. GPUs at FeatureLevel 10.0 and above don't have those strict bytecode limitations and should theoretically fully support SkipOptimization.

The code logic should look something like this:

if (graphicsProfile >= GraphicsProfile.Level_10_0 && isDebugMode) {
    flags |= D3DCOMPILE_SKIP_OPTIMIZATION;
}

3. The Majority of Users Shouldn't Be Sacrificed for a Minority

Here's the reality:

• DirectX 9 is technology from 2002, over 20 years old
• The vast majority of modern games no longer support DirectX 9 (even Unity dropped DX9 support after 2017)
• Stride, as a modern game engine, primarily serves developers creating modern games
• Not being able to disable shader optimization during debugging is a serious developer experience issue

4. Industry Standards

Mainstream engines like Unity and Unreal all provide complete shader optimization control options. This isn't "nice to have" – it's a fundamental feature of professional game engines.

5. Backward Compatibility Shouldn't Come at the Cost of Core Functionality

If the limitations of GraphicsProfile_9_x truly cannot be resolved, then the correct approach should be:

• Provide full debugging support for FeatureLevel 10.0+
• Disable SkipOptimization only for DirectX 9 profile and clearly document the limitations
• Let developers choose whether they need to support this obsolete platform

Rather than preventing all developers using modern GPUs from properly debugging shaders just because we want to support a 20-year-old API.

[Ethereal77]

Ok, my bad. I can make that change and test it. Will do as soon as I can.

Anyway, my post above were not in the sense of "won't do" or "doesn't make sense", just explaining how the Shader Compiler runs currently.

Also, a minor nitpick: GraphicsProfile 9.x is not Direct3D 9, it is a level of functionality equivalent to it, but it also encompasses OpenGL 1.2 to 2.0, OpenGL ES (used for Android until we use Vulkan there), or any other API of that level. In the same way, GraphicsProfile 10 is not Direct3D 10 only, but also OpenGL 3 - 4, etc. This makes more sense than direct equivalence, because if not then what will Vulkan be? It can be 11, but also have features similar to D3D 12. Just thinking out loud 😀

Oh, and btw, D3D9 was developed up until D3D 9.0c in 2004, and D3D 9.x (special version for the Xbox), and D3D10 came with Vista in 2006-2007... so not 20 year old yet 😛

[zgxnet]

Thanks for being open to this! I really appreciate you taking the time to implement and test it. And you're right about the GraphicsProfile distinction – good clarification on the API abstraction layer.

[MsEpsilon]

Wasn't support for DX9 supposed to be dropped for Stride 4.3?

[Kryptos-FR]

From @Ethereal77 earlier:

GraphicsProfile 9.x is not Direct3D 9, it is a level of functionality equivalent to it, but it also encompasses OpenGL 1.2 to 2.0, OpenGL ES (used for Android until we use Vulkan there), or any other API of that level. In the same way, GraphicsProfile 10 is not Direct3D 10 only, but also OpenGL 3 - 4, etc.

[xen2]

I have attached the shader I had issue with.
pixel_shader.txt
I've simplified it as much as I could while still exhibiting the issue.

Here is the fxc command line with /Od (equivalent to skipoptimization).

fxc pixel_shader.txt /T ps_5_0 /E PSMain /Od

internal error: compilation aborted unexpectedly
compilation failed; no code produced

This is the reason why I disabled SkipOptimization during #1715, as I didn't want the shader compiler randomly crashing on us on specific shader.
It's quite subtle, if I do any other change it won't fail anymore.

I think we want to switch from FXC/d3dcompiler_47.dll (obsolete) to DXC/dxcompiler.dll/dxil.dll (maintained).
Maybe somebody more knowledgeable can confirm?

[Ethereal77]

I also suffered from that error while working in the Silk.NET PR. It is quite arbitrary, and could not determine what caused it. I thought it was caused by some kind of memory corruption somewhere, but couldn't verify. I'll test with the shader you uploaded.

With respect to dxcompiler, it's not that simple. DXC, as far as I know, can only produce DXIL (and SPIR-V), in contrast to DirectX bytecode (which is what FXC produces), and DXIL can only be consumed from Direct3D 12 with Shader Model 6.
Actually, it can be used also on Direct3D 11, but not directly. We'd need to implement an abstraction/translation layer using ID3D11On12Device.

[Ethereal77]

I've tested the pixel shader you attached. I have noticed some things that are interesting:

• When compiling with Shader Model ps_2_0, ps_2_a, ps_2_b, or ps_3_0, compilation succeeds, but resulting bytecode is just 1 or 2 instruction slots long.
• When compiling with Shader Model ps_4_0 or higher, it fails with the internal compiler error, and does not generate bytecode.

Claude and GPT said about that that this is a known bug in FXC when working with loops with constant / hardcoded iteration count, and possibly dynamic flow, and it proposed several fixes. From those, two of them compiled succesfully and generated bytecode (but it is 25 or so instruction slots long!).

pixel_shader_fix3.txt

pixel_shader_fix4.txt

The first is just annotating the loop with [unroll] and removing the unused const int. The other one is replacing the if (condition) break; with a if (!condition) { loop_body; }.

Anyway, if it is true that it is a known bug, there's not much we can do except massaging our shaders to compile, I fear. 😣