fix: clean up edge configuration when a project's namespace is purged - #338
Draft
scotwells wants to merge 2 commits into
Draft
fix: clean up edge configuration when a project's namespace is purged#338scotwells wants to merge 2 commits into
scotwells wants to merge 2 commits into
Conversation
… is gone Every downstream cleanup path resolved the edge namespace by reading the upstream Namespace and deriving ns-<uid> from its metadata.uid. A Milo project purge force-finalizes that namespace without waiting for its contents to be cleaned up, so from that point NSO could no longer answer which edge namespace an object belonged to. Its finalizers failed on every attempt, upstream objects stayed terminating forever, and the anchor ConfigMap, mirrored Gateway and DNSEndpoint were stranded on the edge — still propagated to every edge cluster by Karmada, and still serving. When the upstream Namespace is not found, resolve the edge namespace instead from the labels stamped on it at creation time, which identify the upstream cluster and namespace it belongs to. The healthy path is untouched, and because the fallback lives in the shared downstream client, it covers every caller rather than one controller. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VTE43NjiyoTnSn2bAzDS2o
Reproduces the trigger exactly as Milo's project purge performs it: mirror a gateway to the edge, force-finalize the upstream namespace out from under NSO while the gateway is still live, then delete the gateway and require the edge to end up clean. The last step is deliberately not a guarantee. Once a namespace is force-finalized, the API server refuses every write to the objects left inside it, so the upstream gateway's finalizer can never be removed by anyone — the step records that gap and proves it is the API server refusing rather than NSO failing to try. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VTE43NjiyoTnSn2bAzDS2o
scotwells
force-pushed
the
fix/downstream-cleanup-without-upstream-namespace
branch
from
August 6, 2026 02:11
1a000e3 to
14878e5
Compare
6 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
When a customer's project is deleted, the configuration we put on the edge for it can be left behind permanently — the mirrored Gateway, its DNS records, and the anchor that owns them. Karmada propagates those objects out to every edge cluster, so a deleted project's listener keeps serving real traffic with nothing upstream left to explain it or to delete it through. Operators cannot clean it up through the product; it has to be removed by hand on each edge.
The cause is an ordering assumption. NSO works out which edge namespace an object belongs to by reading the customer's namespace in the project control plane. A project purge removes that namespace immediately, without waiting for anything still cleaning up inside it (milo-os/milo#750), so from that moment NSO can no longer answer the question and its cleanup fails on every retry, forever.
This makes cleanup independent of that namespace: when it is gone, the edge namespace is found instead from labels written when it was created. The normal path is unchanged, and because the change is in the shared layer every controller that mirrors resources to the edge is covered, not just gateways.
One half of this stays open and is not ours to close. Once a namespace has been force-finalized, the API server refuses every write to the objects left inside it, so their finalizers can never be removed by anyone — the customer's Gateway object stays in a terminating state until the whole project control plane is torn down. That is the purge-ordering problem in milo-os/milo#750. What reaches customers is the edge, and that is now cleaned up regardless.
Verification
A new end-to-end scenario reproduces the trigger the way a purge performs it: mirror a gateway to the edge, force-finalize the upstream namespace out from under NSO while the gateway is live, delete the gateway, and require the edge to end up clean. It fails against current
mainwith the objects still present and passes with this change. Its last step deliberately records the gap above rather than asserting a guarantee this side of the system cannot keep. The existing project-deletion scenario still passes unchanged.Note for reviewers
The scenario depends on the Milo layer of the prod-fidelity test environment, which is still in flight on a separate branch and not included here. Until that lands the scenario cannot run in CI; the unit tests covering the resolution logic run today.