feat: Skybox light fog

[Eideren]

PR Details

Adds a fog feature to the skybox light, this is different from the fog post process as it matches the skybox used instead of having a singular color used for the whole range.


The idea is to sample the skybox over the surface, increase in mipmap res the further the surface is.

Mar Saba Monastery model belongs to Andrea Spognetta

Related Issue

None.

Types of changes

• Docs change / refactoring / dependency upgrade
• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to change)

Checklist

• My change requires a change to the documentation.
• I have added tests to cover my changes.
• All new and existing tests passed.
• I have built and run the editor to try this change out.

The implementation is kind of hack-y. Hoping Xen2 or Youness could push me in the right direction. On that note, current questions regarding the implementation:

• Does this make sense as part of the Skybox light, we've had some back and forth with Ethereal about this and here's what we think
  • Ethereal is suggesting we move the implementation to be part of the graphics compositor, but with components and parameters still defined in the scene.
  • This implementation is ultimately dependent on the BackgroundComponent's skybox used, not the lights' parameters. So we could split it off of the light either as a new component or something else ?
  • But BackgroundComponent is not guaranteed to be set to a skybox, it could be a 2d texture. We could branch this implementation to handle the different parameters of the background appropriately. But I'm not seeing an obvious way to do so cleanly.
  • To work around the point above, we could create a new component that takes care of both the background and the fog at the same time. Users would remove their background component, and use this one instead. Does that make sense ?
• Should Distance, Curve, NearMip, etc. be generics or leave them as PerView.Lighting ? If they should, how would I set that up.

[github-actions]

🤖 Draft PR — automatic CI is skipped to save runner minutes.

• Mark the PR ready for review to run the full automatic CI — or add a ci-run-on-draft label to run it now without leaving draft.
• Or arm a specific opt-in suite: ci-enduser, ci-editor, ci-ios, ci-android.

[Ethereal77]

I like the effect, but would this not be better as a fog component that samples a texture where you could set there the skybox or any other thing (an arbitrary ComputeColor, for example)?
That way we could have lighting decoupled from volumetrics and still be composable/configurable.

[Eideren]

@Ethereal77 There are two reasons why I moved it to be part of the skybox light:

• The skybox light is there to simulate atmospheric scattering on objects in the scene. This effect is the same idea but on the fine particles standing between you and said object.
• The effect must use the same texture as the one used for the skybox, the illusion breaks as soon as there is a mismatch. Ultimately I would have preferred we made it dependent on the background component, but that one is not guaranteed to be set to a cubemap while the sky light one is...

There's probably a better way to do this, which is why I'm leaving this as draft for now, and would like to hear any further ideas you might have. But I do want to ensure we're not adding more friction for users, it's already a bit stupid to have to swap the texture used in the background component and the skybox asset whenever we want to change the skybox used for a scene. No one in their right mind would keep these distinct, it just looks amateur-ish.

[Ethereal77]

The skybox light is there to simulate atmospheric scattering on objects in the scene

I have always seen the skybox light as just a environment map to use for lighting in PBR, not as an atmospheric scattering simulation, which I see more as "volumetrics".

The effect must use the same texture as the one used for the skybox, the illusion breaks as soon as there is a mismatch.

Yes, I know it would be (and look) weird if the cube texture used is different. I was just talking about differentiating light contribution from volumetric effects.

No one in their right mind would keep these distinct, it just looks amateur-ish.

I myself won't do it normally, and for realistic rendering, but I've seen weird stylistic effects over the years by people that tried weird things no one in their right mind would do 😁

[Eideren]

not as an atmospheric scattering simulation

Simulate as in approximate, like how a skybox approximates the sky by having it painted, or otherwise pre-rendered on a texture to then be projected around the player. Or how an environment map approximates the light an object would receive if it where in the environment depicted in the one captured.

weird stylistic effects over the years by people that tried weird things no one in their right mind would do

Definitely, but I don't think it make sense to have this sort of friction for the 1% that might find it useful. Even more so when you have other ways to do so.

[Ethereal77]

I think you’re misunderstanding my words. What I’m saying is just that imo these two concepts must be separated. I think a light should be just a light (i.e., affect an object's shading as if it was in an environment projecting light on it), and a volumetric effect is a separate concept (i.e., "participating media" simulation and all that, even if only a simple approximation), just like light shafts are implemented today.

I'm not arguing whether it makes more or less sense that the fog and the skybox have the same cube texture. Of course, the most probable use of such a feature would be to have the same texture, I agree.

I'm not opposed to a basic implementation of these "simple" cubemap fog. I may find such an effect useful myself. What I fear is a "hacky" implementation solidifies and later would be difficult to refactor into the good solution if/when we improve how the renderer draws volumetric effects (which we'll need at some point).

[Eideren]

I did understand your position, I was simply clarifying mine. Like I said:

There's probably a better way to do this, which is why I'm leaving this as draft for now, and would like to hear any further ideas you might have.

I think it makes more sense to have such strong dependencies within the same component because of the aforementioned issues with friction on the user side, and redundant logic on the engine side. But it doesn't hold that much water given the background-cubemap/skylight split, and how it really is more dependent on the background itself, so I'm not set either way. 'just want to pick your (and other maintainers) brain on what else we could do, like I mentioned previously.

Should I setup a new component which handles both the 'background' and the fog given that they have to be in sync ?
How could I improve the stuff on the SDSL side, should I setup another collection for enabled fogs ?
Given that it makes sense to limit it to one instance, is there a more efficient way to set it up if we abide by this constraint ?
Are there any features I could add to this which might benefit our users ?

[Ethereal77]

Ok. I dont want`to appear too stubborn on this. I'd also want to hear the opinion of other mantainers.

I'm currently far from my main workstation, so I can't do much. But I'd like to see how light shafts are specifically implemented, to see if the volumetrics part could be expanded and improved, and effects like this one or others could be added.

Should I setup a new component which handles both the 'background' and the fog given that they have to be in sync ?

Is there a way to create a template that adds two components at once? If that were possible, you could keep the two components, but add them and configure them in one go.

Given that it makes sense to limit it to one instance, is there a more efficient way to set it up if we abide by this constraint ?

This is why I talk about a global "volumetrics" pass in ForwardRenderer, which could set a global fog pass in the materials it renders.

[Eideren]

a template that adds two components at once? [...] add them and configure them in one go.

We could but I don't believe this is as much of an issue here; the main thing I'm trying to avoid is having to tweak the background component, the sky light component and the fog component whenever the user wants to swap the skybox. This is a pain in the ass when, for example, one has a bunch of different skyboxes available and is doing look-dev; testing them out to see which one works best, producing multiple versions and comparing them.

Here's what we've discussed, and what I can think of:

• Keep it that way, it's synchronized at least, but not to the right dependency. And for users that don't care about skylight but do want fog, we would have to setup additional logic to avoid the overhead of the skylight's logic.
• Have the fog component reference the background component, but then have to test if the background component is correctly set to a cubemap, and if not, somehow surface that issue to the user in the editor and during asset compilation.
• Have the fog component reference the skybox asset - said skybox asset is not actually related to the background component, it's only used for the skybox light. And so it would only fix this issue in the case where the user has a skybox light as well.
• Have it defined solely within the graphics compositor. Fog is definitely something that is defined on a per-scene basis. For example, it doesn't make sense within a tavern, but it does inside a dank crypt. The background component is also defined within the scene itself, so if we're not doing some lookup to the current scene to find the background and pull the texture from it, our users will have to do it themselves through a component living inside of the scene - and we're effectively back at the initial issue, but with users in the loop.

The above is a response to

This is why I talk about a global "volumetrics" pass in ForwardRenderer, which could set a global fog pass in the materials it renders.

I'm not sure whether you implied that we move all of this to the graphics compositor. I commented on that point none the less as if that was the case.
If the point you were making was about moving the logic to a specific pass, but still have the parameters defined within the scene, I can't discern the difference it would make for the architecture of this feature so I would be fine with it.

[Ethereal77]

Have the fog component reference the background component, but then have to test if the background component is correctly set to a cubemap

I know it would look glaringly wrong in many cases, but for fixed-camera scenes... maybe it would sample the background even if it is a regular 2D texture, or even better, a color or ComputeColor. That would be the most flexible approach, although more complex obviously.

Also, picture another use case I have also seen sometimes. A fake "preview" scene where the background is a static dark image (either 2D or cubemap), but the light is set to a cubemap so the object being previewed at the center looks shaded or lit in a certain way. For example, the typical car selection screen of the Grand Turismo titles.

I'm not sure whether you implied that we move all of this to the graphics compositor. I commented on that point none the less as if that was the case.

Yes, more or less. I was talking more about the future, when we may want to add better volumetric effects.

If it was part of the compositor, maybe a dedicated fog pass (later volumetrics, if we have more) could be configured to sample from several sources:

• None (fixed color, like the fog post-fx we have now, but being applied as a material feature instead of a post-render image effect).
• Image (regular texture).
• Cubemap.
• Background (the scene background, which actually could be an image or a cubemap, but it is taken from there).

Maybe it could default to Background, so your use case always works without setting anything more.

That is more in line with the "volumetrics" idea I had.

[Eideren]

or even better, a color or ComputeColor.

Yeah, we could adapt the implementation used based on whether it's a texture or a cubemap, but we can't use an arbitrary ComputeColor;

• it does not have any scheme for the caller to specify the mip level it should sample at
• it doesn't have an implementation for cubemaps, if there was one, the one to use here would be the projected from view, instead of the more common/expected map to normal or view reflection, and that's something users would have to setup themselves
• we would have to write to the model UV before evaluating it just in case the computecolor used is a texture, which definitely isn't great

picture another use case I have also seen sometimes. A fake "preview" scene where the background is a static dark image[...]

I don't see why they would need a fog whose purpose is specifically to blend into the skybox, and then use a different skybox, they may as well use the simple fog at that point. Even then, they can work around this by setting up a plane without diffuse shading to act as the skybox, or add another background component overriding the first one.

If it was part of the compositor, maybe a dedicated fog pass

We can't have everything setup through the compositor, again, users need to define whether there is fog within a given scene, and what the parameters for this fog would be. We need at least a toggle somewhere on some component or on the scene asset.

[Ethereal77]

Yeah, we could adapt the implementation used based on whether it's a texture or a cubemap, but we can't use an arbitrary ComputeColor;

Well, I said ComputeColor to mean "an arbitrary color we can compute or sample somehow". For something like this we'd need a derived ComputeColor specifically tailored for sampling with direction vectors.

it does not have any scheme for the caller to specify the mip level it should sample at

Currently ComputeTextureBase has no way to specify mip-level, or its Sampler, but it could.

it doesn't have an implementation for cubemaps, if there was one, the one to use here would be the projected from view, instead of the more common/expected map to normal or view reflection

This one I'm not understanding. A ComputeColor tailored to sample from a direction would not care how that direction vector came to be. That is the responsibility of the node calling into the ComputeColor, no? (i.e., in this case the material feature for the fog, I suppose)

we would have to write to the model UV before evaluating it just in case the computecolor used is a texture, which definitely isn't great

I don't understant this one either. Are we talking here about generic "computecolors", or specifically of something else? Because there exists ComputeTexture already and you could say the same about that. Yet exists. For a skybox or a fog that samples a cubemap you don't need a model nor its UVs, just the sampling direction and optionally (as you pointed) a mip-level to sample from. Are you talking about this?

I don't see why they would need a fog whose purpose is specifically to blend into the skybox, and then use a different skybox, they may as well use the simple fog at that point. Even then, they can work around this by setting up a plane without diffuse shading to act as the skybox, or add another background component overriding the first one.

Well, of course you can use a million tricks. But what I'm trying to ideate is an intuitive and flexible way it works most of the time, and one that does not hinder us later if we want to expand this feature.

We can't have everything setup through the compositor, again, users need to define whether there is fog within a given scene, and what the parameters for this fog would be. We need at least a toggle somewhere on some component or on the scene asset.

As I said in another thread (#3299), what I have in mind --even if it is a big reformulation of current rendering-- is a compositor with more flexible configuration of the pipeline (similar to a render graph), instead of the current one with many hardcoded paths and conditions inside ForwardRenderer. In such a scenario, the compositor is the one configuring the rendering passes (for example, fog), and the scene would have components or processors (e.g., a FogComponent in the scene) that merely sets the corresponding flags into its render features in the compositor.

I know this is a big change that we all would need to discuss and design carefuly. But I hope you see where I go with this

[Eideren]

For something like this we'd need a derived ComputeColor specifically tailored for sampling with direction vectors.

Sure, we might need to justify its existence though, that's a lot of additional API for just this feature, ultimately, fairly small feature

A ComputeColor tailored to sample from a direction would not care how that direction vector came to be. That is the responsibility of the node calling into the ComputeColor

Well, I was thinking you wanted to go with using ComputeColor itself, we would build a ComputeCubemap deriving from it directly, and given that ComputeColor is completely opaque, it doesn't have any input the caller can provide, the user would have to set that up on their end when defining the ComputeCubemap with a SampleDirection property set to ViewDirection for example.

Are we talking here about generic "computecolors", or specifically of something else? Because there exists ComputeTexture

A generic ComputeColor, yes. I would rather we didn't branch based on the concrete type. If a user sets up a ComputeTexture as the 'Skybox', ComputeTexture.Compute() will map the texture based on the model's UV. That would not be what we want here, rather, we would want to sample based on the view's horizontal and vertical angle. The only way to do so would be to overwrite the UV before the call, and restore it afterward. Which, like I said, is a bad idea.

Obviously, that doesn't matter if we the base type we accept is not ComputeColor, but something more specific.

and the scene would have components or processors (e.g., a FogComponent in the scene) that merely sets the corresponding flags into its render features in the compositor.

That is fine to me, I should have emphasized that when I meant all of this I was thinking you wanted to remove the component as well.

Although, it would be counterintuitive for users when they try to figure out why their fog component doesn't work, that they have to add the feature to the graphics compositor...

[Ethereal77]

RE: ComputeColor for sampling directions: Sure, we might need to justify its existence though, that's a lot of additional API for just this feature, ultimately, fairly small feature

Well, yes. For this use case of the fog, it's a niche feature. But a ComputeColor-derived type that computes a color based on a direction could be used for many things: procedural sky (from simple gradients à la Minecraft to complex atmospheric scattering simulations), procedural environment mapping (like matcaps or reflection probes), and more.

If a user sets up a ComputeTexture as the 'Skybox', ComputeTexture.Compute() will map the texture based on the model's UV. That would not be what we want here, rather, we would want to sample based on the view's horizontal and vertical angle. The only way to do so would be to overwrite the UV before the call, and restore it afterward. Which, like I said, is a bad idea.

Of course. A ComputeTexture is just a ComputeColor that samples a texture using the stream-provided UVs (usually from the mesh). We'd need something like that, but specific for cubemaps (I think ComputeTexture only supports 2D textures today). This direction would default to the mesh normal, but when composed with other nodes it could receive any direction the composing node wants.
And for this fog use case, the material pipeline would pass the view direction or whatever is appropriate for blending with the background.

But I said above, I think there is value in having a ComputeColor-ish node that receives a direction and produces a color (from which this hypothetical ComputeTextureCube would derive). That way you can have "reflections" that are not from a cubemap, but procedural.

Although, it would be counterintuitive for users when they try to figure out why their fog component doesn't work, that they have to add the feature to the graphics compositor...

Well, the feature can be already there, just as a no-op if no fog is configured. Today, the hardcoded light-shafts path in ForwardRenderer do the same.

[Eideren]

the material pipeline would pass the view direction or whatever is appropriate for blending with the background.

How would you see the SDSL API for that, naively we could introduce float4 Compute(float3 dir, float mipmap) and another signature without mipmap , but it's definitely not as straightforward as ComputeColor's.

Well, the feature can be already there, just as a no-op if no fog is configured. Today, the hardcoded light-shafts path in ForwardRenderer do the same.

Lightshafts are not enabled by default, one has to assign the reference afaict. But yes, we could have the default setup with it on and expect users are not distracted to a point where they forget they disabled it there when wondering why they don't have fog.
I'm wondering whether it even makes sense to have it defined there if the vast majority of the configuration lies in the scene.
We could have a toggle or similar to globally disable it from the compositor, but besides controlling graphics quality by swapping it for the simple fog, I'm not sure what case this would cover. It's both fairly cheap and significant for the mood of a scene, so I'm not sure users would even let their players toggle it off.
What do you think we could do with the compositor or in a future rendergraph if it exists there ? Configuring the order in which 'material features' are drawn could be useful if users want to insert another one prior or after that one, but we don't even have the functionality to support that in SDSL afaict.