Freeze MCPServer generation per pod via downward API

[ChrisJBurns]

Summary

The apply-gate at pkg/container/kubernetes/client.go:530 (PR #5024) was silently defeated as soon as MCPServerGeneration started flowing through the /etc/runconfig/runconfig.json ConfigMap volume — that volume is mounted without subPath, so Kubernetes live-updates it inside every running pod. A restarted old-RS proxyrunner pod (OOM, drain, attach-failure c.exitFunc(1)) re-reads the file after a helm upgrade and picks up the new generation N, while still holding the old image in its CLI args. Both old and new pods then call DeployWorkload with the same ourGen=N, the strict-greater-than gate cannot tell them apart, and the stale-image apply clobbers the fresh one — manifesting as alternating ControllerRevisions and ImagePullBackOff in the bug report.

Freeze MCPServerGeneration per pod via the Kubernetes downward API. The operator stamps toolhive.stacklok.dev/mcpserver-generation: <m.Generation> onto the proxy Deployment's pod template and projects that annotation into the proxyrunner container as the THV_MCPSERVER_GENERATION env var. The proxyrunner uses the env var as an override on the file value, so two coexisting proxyrunner pods carry distinct generations again — symmetric with how the image is already frozen via the CLI positional arg.

Closes #5360.

Medium level

• Shared contract: pkg/container/kubernetes/client.go — new exported constant EnvVarMCPServerGeneration = "THV_MCPSERVER_GENERATION" alongside the existing RunConfigMCPServerGenerationAnnotation. Doc comment on the annotation updated to describe the downward-API projection.
• Operator: cmd/thv-operator/controllers/mcpserver_controller.go — stamps the annotation onto deploymentTemplateAnnotations (next to the existing runconfig-checksum) and injects the env var into the proxyrunner container via valueFrom.fieldRef. deploymentNeedsUpdate mirrors both fields so drift detection stays stable.
• Proxyrunner: cmd/thv-proxyrunner/app/run.go — tryLoadConfigFromFile applies the env-var override on the loaded RunConfig.MCPServerGeneration (warn + fall through on parse error).
• Tests:
  • Lock-in regression test (pkg/container/kubernetes/client_test.go) — two DeployWorkload calls with equal gen + different images; asserts the second clobbers the first. Documents what the gate cannot defend against alone.
  • TDD red→green test (cmd/thv-proxyrunner/app/run_test.go) — confirms env value (3) wins over file value (5) when both are set. Failed before the fix.
  • Downward-API shape test (cmd/thv-operator/controllers/mcpserver_resource_overrides_test.go) — asserts the proxyrunner container declares the env var with ValueFrom.FieldRef.FieldPath pointing at the pod's mcpserver-generation annotation.

Low level

File	Change
pkg/container/kubernetes/client.go	Add EnvVarMCPServerGeneration constant; expand doc comment on RunConfigMCPServerGenerationAnnotation to cover the downward-API projection.
cmd/thv-proxyrunner/app/run.go	Add strconv/kubernetes imports; call new applyMCPServerGenerationOverride after runner.ReadJSON; helper reads the env var, parses as int64, overrides config.MCPServerGeneration.
cmd/thv-operator/controllers/mcpserver_controller.go	Add strconv import. In deploymentForMCPServer: stamp RunConfigMCPServerGenerationAnnotation on deploymentTemplateAnnotations; append the downward-API THV_MCPSERVER_GENERATION env var. Mirror both in deploymentNeedsUpdate so drift comparison stays equal.
cmd/thv-proxyrunner/app/run_test.go	New file. Test TestTryLoadConfigFromFile_MCPServerGenerationEnvOverride.
pkg/container/kubernetes/client_test.go	New test TestDeployWorkload_EqualGenerationDifferentImageClobbers.
cmd/thv-operator/controllers/mcpserver_resource_overrides_test.go	New test TestDeploymentForMCPServer_MCPServerGenerationDownwardAPI. Update existing annotation-count and env-var assertions to account for the new annotation + env var (added at the same relative position in both build and drift paths).

Type of change

• Bug fix
• New feature
• Breaking change
• Refactoring (no behavior change)
• Dependency update
• Documentation
• Other

Test plan

• task test passes (exit 0)
• task build passes
• New unit tests: TestTryLoadConfigFromFile_MCPServerGenerationEnvOverride, TestDeploymentForMCPServer_MCPServerGenerationDownwardAPI, TestDeployWorkload_EqualGenerationDifferentImageClobbers
• Existing drift-detection tests (TestMCPServerDeploymentNeedsUpdate_*) updated and pass — confirms deploymentNeedsUpdate stays in sync with deploymentForMCPServer so reconcile doesn't churn.
• Manual live-cluster reproduction recommended before merge (kind cluster + helm upgrade with maxSurge=1, maxUnavailable=0 + induced container restart on old-RS pod; see issue shouldSkipStatefulSetApply generation gate defeated by live-updating RunConfig ConfigMap during proxy runner rolling update (regression in v0.28.1) #5360 for the exact procedure).

Special notes for reviewers

• Upgrade ordering: the operator and proxyrunner changes are mutually independent during a rolling upgrade. Operator-only or proxyrunner-only upgrades leave the race window open until both sides are at this version — same wire-contract caveat as PR Gate proxyrunner StatefulSet apply by MCPServer generation #5024. Worth calling out in release notes.
• Annotation value "0": when reading existing deployments built by older operators, the pod template won't carry the annotation and the downward-API env var resolves to an empty string. The proxyrunner's applyMCPServerGenerationOverride returns without modifying the config in that case, so the file value wins (existing behavior). After a single reconcile, the operator backfills the annotation and the override kicks in.
• Position of the env var in the container env slice matters — deploymentNeedsUpdate uses equality.Semantic.DeepEqual on the slice. I positioned the new entry in both build sites at the same relative location (after Redis password / user overrides, before embedded auth env vars).
• Out of scope for this PR: the spec.replicas clobber dimension (tracked in [BUG] Improve MCPServer Scaling Coordination #4484) — same root cause family, different field, separate fix.
• The fix path was: investigation → lock-in test for the broken invariant → TDD red test for the env-var override → operator/proxyrunner wiring → drift-detection updates. Each commit compiles and task test passes independently.

Generated with Claude Code

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 68.73%. Comparing base (e37e336) to head (971bbbd).

Additional details and impacted files

@@           Coverage Diff           @@
##             main    #5364   +/-   ##
=======================================
  Coverage   68.72%   68.73%           
=======================================
  Files         626      626           
  Lines       63495    63519   +24     
=======================================
+ Hits        43637    43658   +21     
- Misses      16610    16616    +6     
+ Partials     3248     3245    -3     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[ChrisJBurns]

Nice surgical fix to #5360 — the diagnosis (live-mounted ConfigMap defeating the strict-greater-than gate) is sharp, and the symmetry with how the image is already frozen via the CLI positional arg is the right framing.

The testing pyramid is excellent: lock-in regression (TestDeployWorkload_EqualGenerationDifferentImageClobbers) pinning what the gate cannot defend against, a TDD red→green for the override (TestTryLoadConfigFromFile_MCPServerGenerationEnvOverride), the downward-API shape unit test, and the envtest integration test covering both initial reconcile and spec-bump drift. The defaultAnnotations-takes-precedence merge guarantees user overrides can't accidentally clobber the new annotation.

A couple of small points inline — one defensive-validation nit on the proxyrunner override, and a heads-up that the position-mirror comment in deploymentNeedsUpdate papers over a pre-existing Redis-password drift bug that's worth tracking separately.