fix(llm): zero-downtime LiteLLM config operations — hot-path Reloader stance + x402-buyer split (#321)

.agents/skills/obol-stack-dev/references/llm-routing.md

@@ -24,10 +24,10 @@ Only routes published through Traefik are reachable at `http://obol.stack:8080/`
 |---|---|
 | Traefik ingress (frontend, eRPC, x402 routes) | `http://obol.stack:8080/...` |
 | LiteLLM | `obol kubectl port-forward svc/litellm 14000:4000 -n llm` then `http://127.0.0.1:14000` |
-| x402-buyer sidecar (no Service — pod only) | `obol kubectl port-forward -n llm <litellm-pod> 18402:8402` then `http://127.0.0.1:18402` |
+| x402-buyer (own Deployment + Service) | `obol kubectl port-forward -n llm svc/x402-buyer 18402:8402` then `http://127.0.0.1:18402` |
 | OpenClaw instance | `obol kubectl port-forward -n openclaw-<id> svc/openclaw 18789:18789` |
 
-`http://obol.stack:8080/v1/...` does **not** hit LiteLLM — Traefik has no `/v1` route and returns the frontend 404. The `x402-buyer` sidecar is **distroless** — no `wget`/`curl`/shell. Always port-forward, never `kubectl exec`.
+`http://obol.stack:8080/v1/...` does **not** hit LiteLLM — Traefik has no `/v1` route and returns the frontend 404. The `x402-buyer` binary is **distroless** — no `wget`/`curl`/shell. Always port-forward, never `kubectl exec`.
 
 ## Auto-Configuration During `stack up`
 
@@ -72,16 +72,16 @@ commands directly.
 LiteLLM has a static route added by the embedded config:
 
 ```
-paid/* → openai/* → http://127.0.0.1:8402/v1
+paid/* → openai/* → http://x402-buyer.llm.svc.cluster.local:8402/v1
 ```
 
-The `x402-buyer` sidecar in the litellm pod listens on port 8402.
+`x402-buyer` runs as its own Deployment + Service in the llm namespace, listening on port 8402 (split out of the litellm pod so LiteLLM rolls with maxUnavailable: 0 — issue #321).
 
 **Critical**: the trailing `/v1` is mandatory. LiteLLM's OpenAI provider does **not** append `/v1` to a bare `api_base`. Without it, LiteLLM calls `/chat/completions` on the buyer and the buyer mux returns Go's default `404 page not found`, surfaced as `OpenAIException - 404 page not found`.
 
 ### Buyer flow
 
-`buy.py` (in the agent pod, `${OBOL_SKILLS_DIR:-/data/.openclaw/skills}/buy-x402/scripts/buy.py`) creates a `PurchaseRequest` CR with pre-signed ERC-3009 (USDC) or Permit2 (OBOL) auths. The serviceoffer-controller reconciles the CR, writes per-upstream buyer config/auth files into the buyer ConfigMaps, and hot-adds the `paid/<model>` LiteLLM route. The sidecar spends one auth per paid request.
+`buy.py` (in the agent pod, `${OBOL_SKILLS_DIR:-/data/.openclaw/skills}/buy-x402/scripts/buy.py`) creates a `PurchaseRequest` CR with pre-signed ERC-3009 (USDC) or Permit2 (OBOL) auths. The serviceoffer-controller reconciles the CR, writes per-upstream buyer config/auth files into the buyer ConfigMaps, and hot-adds the `paid/<model>` LiteLLM route. The buyer spends one auth per paid request.
 
 ```
 probe <endpoint-url> [--model <id>]

.agents/skills/obol-stack-dev/references/troubleshooting.md

@@ -61,7 +61,7 @@ obol kubectl create configmap ca-certificates -n x402 \
 
 ### `OpenAIException - 404 page not found` on `paid/<model>`
 
-`api_base` for the buyer route must end in `/v1`. LiteLLM's OpenAI provider does **not** append `/v1` to a bare `api_base`. The buyer route must be `http://127.0.0.1:8402/v1`, not `http://127.0.0.1:8402`.
+`api_base` for the buyer route must end in `/v1`. LiteLLM's OpenAI provider does **not** append `/v1` to a bare `api_base`. The buyer route must be `http://x402-buyer.llm.svc.cluster.local:8402/v1` (the standalone buyer Service), not the bare `:8402` address.
 
 ### `eRPC` `eth_call` cache lag
 

CLAUDE.md

@@ -118,7 +118,7 @@ Payment-gated access to cluster services via x402 (HTTP 402 micropayments, Traef
 
 **Sell flow**: `obol sell http` -> ServiceOffer CR -> controller reconciles: ModelReady -> UpstreamHealthy -> PaymentGateReady (x402 Middleware) -> RoutePublished (HTTPRoute) -> Registered (RegistrationRequest + optional ERC-8004) -> Ready. Traefik routes `/services/<name>/*` through ForwardAuth to upstream.
 
-**Buy flow**: `buy.py probe` sees 402 pricing -> `buy.py buy` validates on-chain token contract -> pre-signs payment auths (ERC-3009 for USDC, Permit2 for OBOL) into `PurchaseRequest` CR in agent ns -> serviceoffer-controller writes buyer config/auth into `llm` ns, publishes `paid/<remote-model>` -> in-pod `x402-buyer` sidecar spends one auth per paid request. Agent-managed refill: `buy.py process --all`, NOT the controller.
+**Buy flow**: `buy.py probe` sees 402 pricing -> `buy.py buy` validates on-chain token contract -> pre-signs payment auths (ERC-3009 for USDC, Permit2 for OBOL) into `PurchaseRequest` CR in agent ns -> serviceoffer-controller writes buyer config/auth into `llm` ns, publishes `paid/<remote-model>` -> the `x402-buyer` Deployment spends one auth per paid request. Agent-managed refill: `buy.py process --all`, NOT the controller.
 
 **buy.py** at `${OBOL_SKILLS_DIR:-/data/.openclaw/skills}/buy-x402/scripts/buy.py` (skill: `buy-x402`, not `buy`):
 ```
@@ -273,14 +273,14 @@ Caveats:
 |---------|---------------------------|
 | Traefik ingress (frontend, eRPC, x402 routes) | `http://obol.stack:8080/...` |
 | LiteLLM (`llm` ns, port 4000) | `kubectl port-forward svc/litellm 14000:4000 -n llm` → `http://127.0.0.1:14000` |
-| x402-buyer sidecar (port 8402, no Service — pod only) | `kubectl port-forward -n llm <litellm-pod> 18402:8402` → `http://127.0.0.1:18402` |
+| x402-buyer (own Deployment + Service, port 8402) | `kubectl port-forward -n llm svc/x402-buyer 18402:8402` → `http://127.0.0.1:18402` |
 | OpenClaw instance | `kubectl port-forward -n openclaw-<id> svc/openclaw 18789:18789` |
 
 **Never call `http://obol.stack:8080/v1/...`** — Traefik has no `/v1` route, returns frontend 404.
 
-**x402-buyer sidecar is distroless** — no `wget`, `curl`, or shell. Use port-forward, not `kubectl exec`.
+**x402-buyer is distroless** — no `wget`, `curl`, or shell. Use port-forward (`svc/x402-buyer`), not `kubectl exec`.
 
-**LiteLLM gateway** (`llm` ns, port 4000): OpenAI-compatible proxy → Ollama/Anthropic/OpenAI. ConfigMap `litellm-config` (YAML config.yaml with model_list), Secret `litellm-secrets` (master key + API keys). Auto-configured with Ollama models during `obol stack up` (no manual `obol model setup`). `ConfigureLiteLLM()` patches config + Secret + restarts or hot-adds via LiteLLM model API. Paid remote inference: Obol LiteLLM fork + `x402-buyer` sidecar, with static `paid/*` → `openai/*` → `http://127.0.0.1:8402` route (wildcard catch-all, requires >=1 concrete `paid/<model>` entry to be useful). Hermes uses provider `"custom"` pointed at `http://litellm.llm.svc.cluster.local:4000/v1`; optional OpenClaw instances reuse the `"openai"` provider slot (ollama slot disabled). Agent configs use `dangerouslyDisableDeviceAuth` for Traefik-proxied access.
+**LiteLLM gateway** (`llm` ns, port 4000): OpenAI-compatible proxy → Ollama/Anthropic/OpenAI. ConfigMap `litellm-config` (YAML config.yaml with model_list), Secret `litellm-secrets` (master key + API keys). Auto-configured with Ollama models during `obol stack up` (no manual `obol model setup`). `ConfigureLiteLLM()` patches config + Secret + restarts or hot-adds via LiteLLM model API. Paid remote inference: Obol LiteLLM fork + standalone `x402-buyer` Deployment, with static `paid/*` → `openai/*` → `http://x402-buyer.llm.svc.cluster.local:8402/v1` route (wildcard catch-all, requires >=1 concrete `paid/<model>` entry to be useful). Hermes uses provider `"custom"` pointed at `http://litellm.llm.svc.cluster.local:4000/v1`; optional OpenClaw instances reuse the `"openai"` provider slot (ollama slot disabled). Agent configs use `dangerouslyDisableDeviceAuth` for Traefik-proxied access.
 
 **Auto-configuration**: `obol stack up` → `autoConfigureLLM()` detects host Ollama models, patches LiteLLM config. `obolup.sh` → `check_agent_model_api_key()` reads `~/.openclaw/openclaw.json`, resolves API key from `ANTHROPIC_API_KEY` / `CLAUDE_CODE_OAUTH_TOKEN` (Anthropic) or `OPENAI_API_KEY` (OpenAI), exports for downstream.
 
@@ -363,9 +363,9 @@ Key code: `internal/inference/{gateway,container,store}.go`, `internal/enclave/{
 
 **Remote-signer wallet**: `GenerateWallet()` in `internal/openclaw/wallet.go`. secp256k1 → Web3 V3 keystore → remote-signer REST API at port 9000 in same ns.
 
-## Buyer Sidecar
+## Buyer (x402-buyer)
 
-`x402-buyer` — lean Go sidecar for buy-side x402 payments using pre-signed ERC-3009 authorizations. Second container in `litellm` Deployment (no separate Service). Agent `buy.py` signs auths locally → `PurchaseRequest`; controller writes per-upstream buyer config/auth files into buyer ConfigMaps, keeps LiteLLM routes in sync. Endpoints: `/status`, `/healthz`, `/metrics`, `/admin/reload`; metrics scraped via `PodMonitor`. Zero signer access, bounded spending (max loss = N × price).
+`x402-buyer` — lean Go service for buy-side x402 payments using pre-signed ERC-3009 authorizations. Standalone Deployment + ClusterIP Service `x402-buyer.llm.svc:8402` (was a litellm sidecar; split out so LiteLLM stays stateless and rolls with `maxUnavailable: 0` — issue #321). Agent `buy.py` signs auths locally → `PurchaseRequest`; controller writes per-upstream buyer config/auth files into buyer ConfigMaps, keeps LiteLLM routes in sync. Endpoints: `/status`, `/healthz`, `/metrics`, `/admin/reload`; metrics scraped via `PodMonitor`. Zero signer access, bounded spending (max loss = N × price).
 
 Settlement lifecycle (cluster-routed paid flow):
 - Traefik/x402-verifier stays on the verify-only path (`verifyOnly: true`).
@@ -414,8 +414,8 @@ A registry digest pin instead of `:latest` on the verifier means your dev rewrit
 3. **ConfigMap propagation** — ~60-120s for k3d file watcher; force restart for immediate effect
 4. **ExternalName services** — do not work with Traefik Gateway API; use ClusterIP + Endpoints
 5. **eRPC `eth_call` cache** — default TTL is 10s for unfinalized reads; `buy.py balance` can lag behind an already-settled paid request for a few seconds
-6. **`/v1` required in `api_base` for `paid/*` route** — LiteLLM's OpenAI provider does NOT append `/v1` to a bare `api_base`. The buyer sidecar route must be `http://127.0.0.1:8402/v1`, not `http://127.0.0.1:8402`. Without `/v1`, LiteLLM calls `/chat/completions` on the buyer; the buyer's mux returns `404 page not found` (Go default), which LiteLLM surfaces as `OpenAIException - 404 page not found`.
-7. **LiteLLM restart is fallback, not the default buy path** — the validated happy path is `buy.py buy`/`process --all`/same-name top-up without a manual LiteLLM restart. Controller hot-add/hot-delete + buyer reload is expected to make `paid/<model>` appear and disappear in place. If a paid alias still fails after controller reconciliation and buyer sidecar reports the upstream, restart LiteLLM as a fallback investigation step. Since rc14, Reloader auto-restarts LiteLLM on `litellm-config` changes (e.g. a buy that adds a NEW provider); buyer CM writes (top-ups/refills) hot-reload via `/admin/reload` with no restart — do not add the buyer CMs to the Reloader annotation.
+6. **`/v1` required in `api_base` for `paid/*` route** — LiteLLM's OpenAI provider does NOT append `/v1` to a bare `api_base`. The buyer route must be `http://x402-buyer.llm.svc.cluster.local:8402/v1`, not `...:8402`. Without `/v1`, LiteLLM calls `/chat/completions` on the buyer; the buyer's mux returns `404 page not found` (Go default), which LiteLLM surfaces as `OpenAIException - 404 page not found`.
+7. **LiteLLM restart is fallback, not the default buy path** — the validated happy path is `buy.py buy`/`process --all`/same-name top-up without a manual LiteLLM restart. Controller hot-add/hot-delete + buyer reload is expected to make `paid/<model>` appear and disappear in place. If a paid alias still fails after controller reconciliation and buyer sidecar reports the upstream, restart LiteLLM as a fallback investigation step. Reloader watches ONLY `litellm-secrets` (key rotation needs a pod replacement; it rolls gaplessly via RollingUpdate maxUnavailable:0). It must NOT watch `litellm-config` — model_list changes are hot-applied via /model/new//model/delete and a CM-triggered rollout would gap inference (issue #321); `obol model status` surfaces CM-vs-router drift if a hot call silently failed. Buyer CM writes (top-ups/refills) hot-reload via `/admin/reload` with no restart — do not add the buyer CMs to any Reloader annotation.
 8. **x402-verifier CA bundle missing → TLS failure** — The `x402-verifier` image is distroless (no CA store). The `ca-certificates` ConfigMap in `x402` namespace must be populated from the host CA bundle or the verifier cannot TLS-verify calls to the facilitator (`https://x402.gcp.obol.tech`), causing `x509: certificate signed by unknown authority` on every payment. **Fixed**: `obol stack up` calls `x402verifier.PopulateCABundle` after infrastructure deployment; `obol sell http` calls it before creating the ServiceOffer. If `Payment verification failed` errors still occur, check verifier logs for the x509 error and repopulate manually: `kubectl create configmap ca-certificates -n x402 --from-file=ca-certificates.crt=/etc/ssl/cert.pem --dry-run=client -o yaml | kubectl replace -f -`
 9. **`EnsureVerifier` overwrites helmfile's image pin under `OBOL_DEVELOPMENT=true`** — `internal/x402/setup.go` reads embedded `x402.yaml` (hard-coded image pin) and `kubectl apply`s it. Without an in-memory rewrite this overwrites the helmfile-managed `:latest` deployment with the embedded pin → every source change to the verifier silently bypassed. Fix shipped in `5a10fb8` (rewrites pins in-memory before apply); structural regression test: `internal/x402/setup_structure_test.go` (`TestEnsureVerifier_NoInlineRegex`). **If you add a new component installed via `kubectl apply` of an embedded manifest**, give it the same dev-rewrite treatment.
 10. **CAIP-2 vs legacy chain id form mismatch** — `RouteRule.Network` is normalized to CAIP-2 (`eip155:84532`) at one boundary, but `internal/x402/chains.go::ResolveChainInfo` must know both that form and the legacy alias (`base-sepolia`). If only one is registered, `matchPaidRouteFull` returns 404 silently on every paid request. When adding a new chain, register both the legacy alias and `CAIP2Network` in every `case` arm.

cmd/obol/model.go

@@ -465,6 +465,12 @@ func modelSetupCustomCommand(cfg *config.Config) *cli.Command {
 type modelStatusResult struct {
 	Providers  []modelStatusProvider `json:"providers"`
 	Discovered []discoverProvider    `json:"discovered,omitempty"`
+	// RouterDrift is set when the live LiteLLM router disagrees with the
+	// litellm-config ConfigMap (a hot-add/hot-delete failed or never ran).
+	// Omitted when the check passed; RouterDriftError explains when the
+	// check itself could not run (cluster or pod unavailable).
+	RouterDrift      *model.RouterDrift `json:"router_drift,omitempty"`
+	RouterDriftError string             `json:"router_drift_error,omitempty"`
 }
 
 type modelStatusProvider struct {
@@ -498,10 +504,21 @@ func modelStatusCommand(cfg *config.Config) *cli.Command {
 			defer cancel()
 			discovered, _ := model.DiscoverLocalProviders(scanCtx)
 
+			// Drift safety net: Reloader no longer restarts LiteLLM on
+			// litellm-config changes, so a silently-failed hot-add/hot-delete
+			// is only visible here. Non-fatal — status must still render
+			// when the cluster or pod is unreachable.
+			drift, driftErr := model.CheckRouterDrift(cfg)
+
 			if u.IsJSON() {
 				result := modelStatusResult{
 					Discovered: discoveredProvidersToJSON(discovered),
 				}
+				if driftErr != nil {
+					result.RouterDriftError = driftErr.Error()
+				} else if !drift.Empty() {
+					result.RouterDrift = &drift
+				}
 				for _, name := range providers {
 					s := status[name]
 					key := "n/a"
@@ -548,6 +565,21 @@ func modelStatusCommand(cfg *config.Config) *cli.Command {
 			}
 
 			u.Blank()
+			switch {
+			case driftErr != nil:
+				u.Dim(fmt.Sprintf("  Live router check unavailable: %v", driftErr))
+				u.Blank()
+			case !drift.Empty():
+				u.Warnf("LiteLLM live router disagrees with litellm-config (hot update failed?):")
+				for _, m := range drift.Missing {
+					u.Printf("    missing from router: %s (requests 404 until next LiteLLM rollout)", m)
+				}
+				for _, m := range drift.Extra {
+					u.Printf("    extra in router:     %s (disappears on next LiteLLM rollout)", m)
+				}
+				u.Dim("  Fix: `obol kubectl rollout restart deployment/litellm -n llm` reloads the ConfigMap.")
+				u.Blank()
+			}
 			if printModelRanking(u, cfg, "section") {
 				u.Blank()
 			}

flows/buy-external.sh

@@ -222,7 +222,7 @@ external_snapshot_on_fail() {
         python3 -c "
 import urllib.request, json
 try:
-    resp = urllib.request.urlopen('http://localhost:8402/status', timeout=5)
+    resp = urllib.request.urlopen('http://x402-buyer.llm.svc.cluster.local:8402/status', timeout=5)
     print(json.dumps(json.loads(resp.read()), indent=2))
 except Exception as e:
     print(json.dumps({'error': repr(e)}))
@@ -478,7 +478,7 @@ bob kubectl exec -n llm deployment/litellm -c litellm -- \
     python3 -c "
 import urllib.request, json, sys
 try:
-    resp = urllib.request.urlopen('http://localhost:8402/status', timeout=5)
+    resp = urllib.request.urlopen('http://x402-buyer.llm.svc.cluster.local:8402/status', timeout=5)
     print(json.dumps(json.loads(resp.read()), indent=2))
 except Exception as e:
     print(json.dumps({'error': repr(e)}))
@@ -550,7 +550,7 @@ bob kubectl rollout status deployment/litellm -n llm --timeout=180s >/dev/null 2
 sidecar_status_raw=$(bob kubectl exec -n llm deployment/litellm -c litellm -- \
     python3 -c "
 import urllib.request
-print(urllib.request.urlopen('http://localhost:8402/status', timeout=5).read().decode())
+print(urllib.request.urlopen('http://x402-buyer.llm.svc.cluster.local:8402/status', timeout=5).read().decode())
 " 2>/dev/null || true)
 PAID_MODEL=$(printf '%s' "$sidecar_status_raw" | python3 -c "
 import json, sys
@@ -682,7 +682,7 @@ bob kubectl exec -n llm deployment/litellm -c litellm -- \
     python3 -c "
 import urllib.request, json
 try:
-    resp = urllib.request.urlopen('http://localhost:8402/status', timeout=5)
+    resp = urllib.request.urlopen('http://x402-buyer.llm.svc.cluster.local:8402/status', timeout=5)
     print(json.dumps(json.loads(resp.read()), indent=2))
 except Exception as e:
     print(json.dumps({'error': repr(e)}))