Fix depth in SDSM shadows

[jeannekamikaze]

This fixes #2620, though I am sure not in the cleanest way. Please provide feedback or feel free to throw this whole PR away.

#2620 shows a sample app for the SDSM filter that also uses multisampling. SDSM was not written with multisampling in mind. This change makes SDSM work with and without it.

As I explained in the issue, the problem is that the depth texture that the SDSM uses becomes multisampled, so texture2D becomes texture2DMS, and the texture cannot be sampled but only fetched. Moreover, the logic in the FilterPostProcessor to handle multisampling mostly deals with materials, which the two compute shaders in SDSM are agnostic of.

The change is to have SdsmFitter check whether the input depth texture is multisampled and compile the two shaders accordingly. This introduces some CPU overhead but is the simplest solution without having to track and maintain additional state.

The crux of the change is in MultiSample.glsllib. I have added additional overloads there to work with/out multisampled textures.

I think there are still three problems not addressed here, though:

• The existing getDepth(in sampler2DMS tex, in vec2 texC) is kind of cooked. It calls textureFetch(), which is doing an average of all MS samples. This is rarely what you want for depth. Specifically, at an edge between geometry and background, it'll average values that range between "foreground" and "background" and give something in between. Rather, for depth, you typically want either a max or a min of the samples, depending on the algorithm. Here's a random post about this: https://wickedengine.net/2016/11/how-to-resolve-an-msaa-depthbuffer/ For SDSM in particular, I went with max. I also did not want to change the existing getDepth() because that might break a whole bunch of other shaders that I have not tested.
• Going with max with SDSM may defeat the purpose of multisampling a bit. Like I mentioned in Setting 'setNumSamples' on FilterPostProcessor causes SdsmDirectionalLightShadowFilter to crash. #2620, for some shadow algos it's often better to run the algo on each sample and then average the results. This typically gives a smoother shadow. However, that would also increase the cost of the shader by m_NumDepthSamples. The max should be just as bad as in the no-multisampling case, but not worse. I'm also not entirely familiar with SDSM anyway.
• I'm not really sure it should be up to SdsmFitter and individual compute shader-based post-filters to deal with multisampling. I think this would ideally be abstracted away, either by having the FilterPostProcessor manage compute shaders better, or by resolving the MS textures before the filters are called. But this would require a larger change and more discussion.

Testing

I have tested the sample app in the issue with and without the line fpp.setNumSamples(2).

[codex128]

Unrelated to your changes, but I found a bug while testing this PR. #2631

[jeannekamikaze]

Hello, thank you. Will address your comments soon. I am a Java noob and admittedly should have done a better self-review.

[codex128]

Looks good. Thanks for the contribution!

[riccardobl]

Thank you, there are some changes that are required to get this well integrated into the engine.
Can you look into them?

[riccardobl]

@gemini-code-assist please review

[gemini-code-assist]

Code Review

This pull request introduces support for multisampling in SDSM shadows by creating shader variants using preprocessor defines. The changes in SdsmFitter to manage these shader variants are well-structured. The new ComputeShader constructor for adding defines is a good addition, though it has a potential ArrayIndexOutOfBoundsException that should be addressed. I've also found a critical issue in SdsmPostShadow.frag where texture sampling functions appear to be swapped, which would lead to incorrect rendering. Overall, this is a solid approach to the problem, with a couple of important fixes needed.