Hide Sentinel.UNSET values as None in lookup_default()

[kdeldycke]

This PR address issue #3145, in which the UNSET sentinel is leaking into lookup_default() when users have overriding it.

This changes prevent lookup_default() to return UNSET as-is, and masquerade it as None. To account for this change in lookup_default() behavior, I had to update some internal checks around defaults.

Additionally, to prevent any side-effects, I added a couple of tests to fill the gaps in coverage of lookup_default(), especially around treatment of absence of value and value of absence.

Important

I did use LLMs (Claude Opus 4.6) in the process of creating this PR, but as a co-worker to criticize my changes and modifications, not as a vibe-coding agent. This PR was carefully written, edited and reviewed by me.

[kdeldycke]

This PR replace AI-slop from: #3199, #3202, #3209, #3212 and #3215.

[kdeldycke]

Also note that the unnatural None dance around the results of lookup_default were made to please mypy.

[davidism]

This is fine for the bug fix, but you're right that this is getting messy. We need to figure out a long term solution.

[kdeldycke]

@Rowlando13, should we merge? to main or stable?

[Rowlando13]

Main. @kdeldycke Getting this fix is good, but it would be better if we try to make larger fixes to simplify the issues Click has accumulated.

[Rowlando13]

9.0 should be next release.

[kdeldycke]

Looks like @davidism is in favor on Discord of getting 8.3.2 out first, which could include this fix, which as limited blast radius. 😁

[kdeldycke]

FYI, just tested this fix against my own Click Extra codebase and could not find any regression: kdeldycke/click-extra@4bbd387

[Rowlando13]

Sounds good.

[kdeldycke]

Thanks @Rowlando13 for the merge!

[kdeldycke]

UNSET is not a public API, so I'm not clear why you'd use that. Don't add the key to the default map if you don't want to change the default.

This is a reasonable assumption as in dict, you can materialize the absence of value by the absence of the key. But even if @sankarcn is a bot, it can still point to a gap in the tests around default_map. I will explore that in #3240

[sankarcn]

@kdeldycke I am a human. Sorry, I could not reply to you and considered your reply as a guidance and I did not have anything to add. This was one of the edge cases, I came across when testing the PR, for a different purpose.

[kdeldycke]

@kdeldycke I am a human. Sorry, I could not reply to you and considered your reply as a guidance and I did not have anything to add. This was one of the edge cases, I came across when testing the PR, for a different purpose.

Ahah sorry, paranoia is high these days among open-source maintainers. Maybe you can tell us more about this use case. If you stumble upon it, there is a high chance others will. So I'd like to make a case so we can reduce the blast-radius of UNSET. Let's discuss this at: #3240