fix(llm): zero-downtime LiteLLM config operations — hot-path Reloader stance + x402-buyer split (#321)

[bussyjd]

LiteLLM zero-downtime config operations (#321 phases 1+2)

Problem

The hot /model/new path shipped in #320 was being defeated by two later changes:

1. 2eed34b added a Reloader annotation on litellm-config. Every model add/remove/prefer and every first-time purchase patches that ConfigMap (persistence-first ordering), so Reloader rolled the pod seconds after the hot-add succeeded.
2. 5c9a879 moved the buyer's consumed-auth state to an RWO PVC inside the litellm pod, forcing replicas: 1 + strategy: Recreate — every rollout became a full inference gap.

Net effect: every config operation took the inference gateway down, which is exactly what #321 set out to fix (and what the latest issue comment asks about).

Phase 1 — stop self-inflicted restarts

• Reloader now watches litellm-secrets only. litellm-config changes are hot-applied via /model/new + /model/delete by both the CLI and the serviceoffer-controller; the ConfigMap is persistence for the next pod start.
• obol model prefer no longer restarts LiteLLM (order is an obol convention read from the ConfigMap by Rank(); LiteLLM's router ignores it).
• RestartLiteLLM fails loudly on rollout timeout instead of warning and reporting success (LiteLLM reliability: hot-reload, zero-downtime restarts, and single-replica fragility #321 item 3).
• Drift safety net replacing the Reloader annotation: obol model status now compares the ConfigMap model_list against the live router /v1/models and reports missing/extra entries (internal/model/drift.go), so a silently-failed hot call is visible instead of masked by a restart.
• ReconcileRecorded's ConfigMap-only branch restarts explicitly now that Reloader won't.

Phase 2 — make the remaining restarts gapless

• x402-buyer is split out of the litellm pod into its own Deployment (1 replica, Recreate, keeps the RWO PVC and single-writer auth semantics) + ClusterIP Service x402-buyer.llm.svc:8402.
• litellm is now stateless → RollingUpdate maxSurge: 1, maxUnavailable: 0. Secret rotation (via Reloader) and image bumps surge a new pod to Ready before the old one terminates — zero inference gap. Still replicas: 1 steady-state (no extra laptop footprint).
• Paid-route api_base → http://x402-buyer.llm.svc.cluster.local:8402/v1 (chart wildcard + controller-written entries).
• Upgrade path: the controller migrates legacy 127.0.0.1:8402 entries at startup and on per-purchase reconcile (ConfigMap rewrite + hot delete/re-add). Those entries are dead upstreams on upgraded clusters until migrated, so the delete/re-add window is not a regression.
• Controller buyer probes (/admin/reload, /admin/remove, /status) and agent buy.py target app=x402-buyer pods; flow scripts reach the buyer via its Service.
• PodMonitor keeps its historical name (litellm-x402-buyer) but targets the buyer pods.

Deliberately deferred

• NetworkPolicy on :8402 — agent namespaces (hermes-obol-agent, openclaw-<id>, agent CRs) share no common label and buy.py legitimately reads /status from them. The Service changes addressing, not exposure (the pod IP was already cluster-reachable). Follow-up: label agent namespaces, then restrict.
• Hot API-key rotation (fork /model/update with in-memory key) — unnecessary now that key rotation rolls gaplessly; would only remove the surge itself.
• Buyer HA (auth-pool sharding) — buyer rollouts are rare (image bumps only; CM changes hot-reload) and brief.

Review boundaries / invariants

• x402-buyer must have exactly one writer of consumed-auth state: buyer Deployment is replicas: 1 + Recreate, PVC unchanged.
• litellm Deployment must mount no PVC and carry no buyer container (pinned by TestBuyerStatePVC).
• Reloader annotation must not include litellm-config (pinned by TestLLMTemplate_IncludesPaidRouteAndBuyerSidecar).
• Controller must never fall back to a pod restart on hot-add failure (unchanged behavior, addLiteLLMModelEntry).
• /v1 suffix on buyer api_base is load-bearing (CLAUDE.md pitfall 6; pinned by TestBuyerAPIBase).
• No changes to payment verification, settlement, RBAC, or route exposure.

Tests

• internal/model/drift_test.go — table-driven DiffRouterModels (wildcards, missing, extra).
• internal/serviceoffercontroller/purchase_migration_test.go — legacy api_base migration (startup + per-purchase + no-op + missing-CM), buyerAPIBase.
• Updated structural pins: internal/stack/stack_test.go (annotation stance, service api_base), internal/embed/embed_buyer_state_test.go (buyer Deployment owns PVC, litellm stateless + RollingUpdate params).
• Full go test ./... green.

Local validation (fresh k3d cluster, OBOL_DEVELOPMENT=true, dev-built CLI + controller image)

An in-cluster probe hit litellm.llm.svc:4000/health/readiness twice a second for the whole session: 1253 samples, 0 failures across all of the following.

1. Topology: litellm pod is single-container, RollingUpdate {maxSurge:1, maxUnavailable:0}, annotations carry secret.reloader.stakater.com/reload only; x402-buyer runs as its own 1-replica Recreate Deployment with the PVC, plus ClusterIP Service.
2. obol model prefer qwen3.5 → ConfigMap reordered, same pod, restartCount 0.
3. obol model remove nomic-embed-text → hot-delete, same pod; obol model status drift check stayed clean (live router really dropped it).
4. Drift net: appending a bogus entry straight to the CM (bypassing the hot API) made obol model status print missing from router: drift-test-ghost with the fix hint; restoring the CM cleared it.
5. Secret rotation: patching litellm-secrets triggered a Reloader surge rollout — old and new pods briefly coexisted, buyer pod untouched, zero probe failures through the replacement.
6. Buyer via Service: /healthz → ok, /status → {}.
7. Upgrade migration, live: injected a paid/zdt-test entry with the legacy 127.0.0.1:8402/v1 api_base, restarted the controller → log buyer-migrate: rewrote 1 legacy buyer api_base entries in llm/litellm-config, CM rewritten to the Service URL, entry hot-added.
8. End-to-end path: calling paid/zdt-test through LiteLLM returned the buyer's own application error (no purchased upstream mapped for requested model) — LiteLLM → x402-buyer Service connectivity confirmed; only an actual purchase is absent.
9. Cleanup verified: obol model remove paid/zdt-test (live + config), final drift check clean.

Not exercised live: a real paid buy (needs funded wallet + seller; covered by release-gate flows 06/08/11/13/14, which this PR updates for the Service address).