diff --git a/.agents/skills/obol-stack-dev/references/llm-routing.md b/.agents/skills/obol-stack-dev/references/llm-routing.md index 60b5d530..d2abac58 100644 --- a/.agents/skills/obol-stack-dev/references/llm-routing.md +++ b/.agents/skills/obol-stack-dev/references/llm-routing.md @@ -24,10 +24,10 @@ Only routes published through Traefik are reachable at `http://obol.stack:8080/` |---|---| | Traefik ingress (frontend, eRPC, x402 routes) | `http://obol.stack:8080/...` | | LiteLLM | `obol kubectl port-forward svc/litellm 14000:4000 -n llm` then `http://127.0.0.1:14000` | -| x402-buyer sidecar (no Service — pod only) | `obol kubectl port-forward -n llm 18402:8402` then `http://127.0.0.1:18402` | +| x402-buyer (own Deployment + Service) | `obol kubectl port-forward -n llm svc/x402-buyer 18402:8402` then `http://127.0.0.1:18402` | | OpenClaw instance | `obol kubectl port-forward -n openclaw- svc/openclaw 18789:18789` | -`http://obol.stack:8080/v1/...` does **not** hit LiteLLM — Traefik has no `/v1` route and returns the frontend 404. The `x402-buyer` sidecar is **distroless** — no `wget`/`curl`/shell. Always port-forward, never `kubectl exec`. +`http://obol.stack:8080/v1/...` does **not** hit LiteLLM — Traefik has no `/v1` route and returns the frontend 404. The `x402-buyer` binary is **distroless** — no `wget`/`curl`/shell. Always port-forward, never `kubectl exec`. ## Auto-Configuration During `stack up` @@ -72,16 +72,16 @@ commands directly. LiteLLM has a static route added by the embedded config: ``` -paid/* → openai/* → http://127.0.0.1:8402/v1 +paid/* → openai/* → http://x402-buyer.llm.svc.cluster.local:8402/v1 ``` -The `x402-buyer` sidecar in the litellm pod listens on port 8402. +`x402-buyer` runs as its own Deployment + Service in the llm namespace, listening on port 8402 (split out of the litellm pod so LiteLLM rolls with maxUnavailable: 0 — issue #321). **Critical**: the trailing `/v1` is mandatory. LiteLLM's OpenAI provider does **not** append `/v1` to a bare `api_base`. Without it, LiteLLM calls `/chat/completions` on the buyer and the buyer mux returns Go's default `404 page not found`, surfaced as `OpenAIException - 404 page not found`. ### Buyer flow -`buy.py` (in the agent pod, `${OBOL_SKILLS_DIR:-/data/.openclaw/skills}/buy-x402/scripts/buy.py`) creates a `PurchaseRequest` CR with pre-signed ERC-3009 (USDC) or Permit2 (OBOL) auths. The serviceoffer-controller reconciles the CR, writes per-upstream buyer config/auth files into the buyer ConfigMaps, and hot-adds the `paid/` LiteLLM route. The sidecar spends one auth per paid request. +`buy.py` (in the agent pod, `${OBOL_SKILLS_DIR:-/data/.openclaw/skills}/buy-x402/scripts/buy.py`) creates a `PurchaseRequest` CR with pre-signed ERC-3009 (USDC) or Permit2 (OBOL) auths. The serviceoffer-controller reconciles the CR, writes per-upstream buyer config/auth files into the buyer ConfigMaps, and hot-adds the `paid/` LiteLLM route. The buyer spends one auth per paid request. ``` probe [--model ] diff --git a/.agents/skills/obol-stack-dev/references/troubleshooting.md b/.agents/skills/obol-stack-dev/references/troubleshooting.md index 2d24218c..00a4121b 100644 --- a/.agents/skills/obol-stack-dev/references/troubleshooting.md +++ b/.agents/skills/obol-stack-dev/references/troubleshooting.md @@ -61,7 +61,7 @@ obol kubectl create configmap ca-certificates -n x402 \ ### `OpenAIException - 404 page not found` on `paid/` -`api_base` for the buyer route must end in `/v1`. LiteLLM's OpenAI provider does **not** append `/v1` to a bare `api_base`. The buyer route must be `http://127.0.0.1:8402/v1`, not `http://127.0.0.1:8402`. +`api_base` for the buyer route must end in `/v1`. LiteLLM's OpenAI provider does **not** append `/v1` to a bare `api_base`. The buyer route must be `http://x402-buyer.llm.svc.cluster.local:8402/v1` (the standalone buyer Service), not the bare `:8402` address. ### `eRPC` `eth_call` cache lag diff --git a/CLAUDE.md b/CLAUDE.md index 3eaf1db6..ef64a7f7 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -118,7 +118,7 @@ Payment-gated access to cluster services via x402 (HTTP 402 micropayments, Traef **Sell flow**: `obol sell http` -> ServiceOffer CR -> controller reconciles: ModelReady -> UpstreamHealthy -> PaymentGateReady (x402 Middleware) -> RoutePublished (HTTPRoute) -> Registered (RegistrationRequest + optional ERC-8004) -> Ready. Traefik routes `/services//*` through ForwardAuth to upstream. -**Buy flow**: `buy.py probe` sees 402 pricing -> `buy.py buy` validates on-chain token contract -> pre-signs payment auths (ERC-3009 for USDC, Permit2 for OBOL) into `PurchaseRequest` CR in agent ns -> serviceoffer-controller writes buyer config/auth into `llm` ns, publishes `paid/` -> in-pod `x402-buyer` sidecar spends one auth per paid request. Agent-managed refill: `buy.py process --all`, NOT the controller. +**Buy flow**: `buy.py probe` sees 402 pricing -> `buy.py buy` validates on-chain token contract -> pre-signs payment auths (ERC-3009 for USDC, Permit2 for OBOL) into `PurchaseRequest` CR in agent ns -> serviceoffer-controller writes buyer config/auth into `llm` ns, publishes `paid/` -> the `x402-buyer` Deployment spends one auth per paid request. Agent-managed refill: `buy.py process --all`, NOT the controller. **buy.py** at `${OBOL_SKILLS_DIR:-/data/.openclaw/skills}/buy-x402/scripts/buy.py` (skill: `buy-x402`, not `buy`): ``` @@ -273,14 +273,14 @@ Caveats: |---------|---------------------------| | Traefik ingress (frontend, eRPC, x402 routes) | `http://obol.stack:8080/...` | | LiteLLM (`llm` ns, port 4000) | `kubectl port-forward svc/litellm 14000:4000 -n llm` → `http://127.0.0.1:14000` | -| x402-buyer sidecar (port 8402, no Service — pod only) | `kubectl port-forward -n llm 18402:8402` → `http://127.0.0.1:18402` | +| x402-buyer (own Deployment + Service, port 8402) | `kubectl port-forward -n llm svc/x402-buyer 18402:8402` → `http://127.0.0.1:18402` | | OpenClaw instance | `kubectl port-forward -n openclaw- svc/openclaw 18789:18789` | **Never call `http://obol.stack:8080/v1/...`** — Traefik has no `/v1` route, returns frontend 404. -**x402-buyer sidecar is distroless** — no `wget`, `curl`, or shell. Use port-forward, not `kubectl exec`. +**x402-buyer is distroless** — no `wget`, `curl`, or shell. Use port-forward (`svc/x402-buyer`), not `kubectl exec`. -**LiteLLM gateway** (`llm` ns, port 4000): OpenAI-compatible proxy → Ollama/Anthropic/OpenAI. ConfigMap `litellm-config` (YAML config.yaml with model_list), Secret `litellm-secrets` (master key + API keys). Auto-configured with Ollama models during `obol stack up` (no manual `obol model setup`). `ConfigureLiteLLM()` patches config + Secret + restarts or hot-adds via LiteLLM model API. Paid remote inference: Obol LiteLLM fork + `x402-buyer` sidecar, with static `paid/*` → `openai/*` → `http://127.0.0.1:8402` route (wildcard catch-all, requires >=1 concrete `paid/` entry to be useful). Hermes uses provider `"custom"` pointed at `http://litellm.llm.svc.cluster.local:4000/v1`; optional OpenClaw instances reuse the `"openai"` provider slot (ollama slot disabled). Agent configs use `dangerouslyDisableDeviceAuth` for Traefik-proxied access. +**LiteLLM gateway** (`llm` ns, port 4000): OpenAI-compatible proxy → Ollama/Anthropic/OpenAI. ConfigMap `litellm-config` (YAML config.yaml with model_list), Secret `litellm-secrets` (master key + API keys). Auto-configured with Ollama models during `obol stack up` (no manual `obol model setup`). `ConfigureLiteLLM()` patches config + Secret + restarts or hot-adds via LiteLLM model API. Paid remote inference: Obol LiteLLM fork + standalone `x402-buyer` Deployment, with static `paid/*` → `openai/*` → `http://x402-buyer.llm.svc.cluster.local:8402/v1` route (wildcard catch-all, requires >=1 concrete `paid/` entry to be useful). Hermes uses provider `"custom"` pointed at `http://litellm.llm.svc.cluster.local:4000/v1`; optional OpenClaw instances reuse the `"openai"` provider slot (ollama slot disabled). Agent configs use `dangerouslyDisableDeviceAuth` for Traefik-proxied access. **Auto-configuration**: `obol stack up` → `autoConfigureLLM()` detects host Ollama models, patches LiteLLM config. `obolup.sh` → `check_agent_model_api_key()` reads `~/.openclaw/openclaw.json`, resolves API key from `ANTHROPIC_API_KEY` / `CLAUDE_CODE_OAUTH_TOKEN` (Anthropic) or `OPENAI_API_KEY` (OpenAI), exports for downstream. @@ -363,9 +363,9 @@ Key code: `internal/inference/{gateway,container,store}.go`, `internal/enclave/{ **Remote-signer wallet**: `GenerateWallet()` in `internal/openclaw/wallet.go`. secp256k1 → Web3 V3 keystore → remote-signer REST API at port 9000 in same ns. -## Buyer Sidecar +## Buyer (x402-buyer) -`x402-buyer` — lean Go sidecar for buy-side x402 payments using pre-signed ERC-3009 authorizations. Second container in `litellm` Deployment (no separate Service). Agent `buy.py` signs auths locally → `PurchaseRequest`; controller writes per-upstream buyer config/auth files into buyer ConfigMaps, keeps LiteLLM routes in sync. Endpoints: `/status`, `/healthz`, `/metrics`, `/admin/reload`; metrics scraped via `PodMonitor`. Zero signer access, bounded spending (max loss = N × price). +`x402-buyer` — lean Go service for buy-side x402 payments using pre-signed ERC-3009 authorizations. Standalone Deployment + ClusterIP Service `x402-buyer.llm.svc:8402` (was a litellm sidecar; split out so LiteLLM stays stateless and rolls with `maxUnavailable: 0` — issue #321). Agent `buy.py` signs auths locally → `PurchaseRequest`; controller writes per-upstream buyer config/auth files into buyer ConfigMaps, keeps LiteLLM routes in sync. Endpoints: `/status`, `/healthz`, `/metrics`, `/admin/reload`; metrics scraped via `PodMonitor`. Zero signer access, bounded spending (max loss = N × price). Settlement lifecycle (cluster-routed paid flow): - Traefik/x402-verifier stays on the verify-only path (`verifyOnly: true`). @@ -414,8 +414,8 @@ A registry digest pin instead of `:latest` on the verifier means your dev rewrit 3. **ConfigMap propagation** — ~60-120s for k3d file watcher; force restart for immediate effect 4. **ExternalName services** — do not work with Traefik Gateway API; use ClusterIP + Endpoints 5. **eRPC `eth_call` cache** — default TTL is 10s for unfinalized reads; `buy.py balance` can lag behind an already-settled paid request for a few seconds -6. **`/v1` required in `api_base` for `paid/*` route** — LiteLLM's OpenAI provider does NOT append `/v1` to a bare `api_base`. The buyer sidecar route must be `http://127.0.0.1:8402/v1`, not `http://127.0.0.1:8402`. Without `/v1`, LiteLLM calls `/chat/completions` on the buyer; the buyer's mux returns `404 page not found` (Go default), which LiteLLM surfaces as `OpenAIException - 404 page not found`. -7. **LiteLLM restart is fallback, not the default buy path** — the validated happy path is `buy.py buy`/`process --all`/same-name top-up without a manual LiteLLM restart. Controller hot-add/hot-delete + buyer reload is expected to make `paid/` appear and disappear in place. If a paid alias still fails after controller reconciliation and buyer sidecar reports the upstream, restart LiteLLM as a fallback investigation step. Since rc14, Reloader auto-restarts LiteLLM on `litellm-config` changes (e.g. a buy that adds a NEW provider); buyer CM writes (top-ups/refills) hot-reload via `/admin/reload` with no restart — do not add the buyer CMs to the Reloader annotation. +6. **`/v1` required in `api_base` for `paid/*` route** — LiteLLM's OpenAI provider does NOT append `/v1` to a bare `api_base`. The buyer route must be `http://x402-buyer.llm.svc.cluster.local:8402/v1`, not `...:8402`. Without `/v1`, LiteLLM calls `/chat/completions` on the buyer; the buyer's mux returns `404 page not found` (Go default), which LiteLLM surfaces as `OpenAIException - 404 page not found`. +7. **LiteLLM restart is fallback, not the default buy path** — the validated happy path is `buy.py buy`/`process --all`/same-name top-up without a manual LiteLLM restart. Controller hot-add/hot-delete + buyer reload is expected to make `paid/` appear and disappear in place. If a paid alias still fails after controller reconciliation and buyer sidecar reports the upstream, restart LiteLLM as a fallback investigation step. Reloader watches ONLY `litellm-secrets` (key rotation needs a pod replacement; it rolls gaplessly via RollingUpdate maxUnavailable:0). It must NOT watch `litellm-config` — model_list changes are hot-applied via /model/new//model/delete and a CM-triggered rollout would gap inference (issue #321); `obol model status` surfaces CM-vs-router drift if a hot call silently failed. Buyer CM writes (top-ups/refills) hot-reload via `/admin/reload` with no restart — do not add the buyer CMs to any Reloader annotation. 8. **x402-verifier CA bundle missing → TLS failure** — The `x402-verifier` image is distroless (no CA store). The `ca-certificates` ConfigMap in `x402` namespace must be populated from the host CA bundle or the verifier cannot TLS-verify calls to the facilitator (`https://x402.gcp.obol.tech`), causing `x509: certificate signed by unknown authority` on every payment. **Fixed**: `obol stack up` calls `x402verifier.PopulateCABundle` after infrastructure deployment; `obol sell http` calls it before creating the ServiceOffer. If `Payment verification failed` errors still occur, check verifier logs for the x509 error and repopulate manually: `kubectl create configmap ca-certificates -n x402 --from-file=ca-certificates.crt=/etc/ssl/cert.pem --dry-run=client -o yaml | kubectl replace -f -` 9. **`EnsureVerifier` overwrites helmfile's image pin under `OBOL_DEVELOPMENT=true`** — `internal/x402/setup.go` reads embedded `x402.yaml` (hard-coded image pin) and `kubectl apply`s it. Without an in-memory rewrite this overwrites the helmfile-managed `:latest` deployment with the embedded pin → every source change to the verifier silently bypassed. Fix shipped in `5a10fb8` (rewrites pins in-memory before apply); structural regression test: `internal/x402/setup_structure_test.go` (`TestEnsureVerifier_NoInlineRegex`). **If you add a new component installed via `kubectl apply` of an embedded manifest**, give it the same dev-rewrite treatment. 10. **CAIP-2 vs legacy chain id form mismatch** — `RouteRule.Network` is normalized to CAIP-2 (`eip155:84532`) at one boundary, but `internal/x402/chains.go::ResolveChainInfo` must know both that form and the legacy alias (`base-sepolia`). If only one is registered, `matchPaidRouteFull` returns 404 silently on every paid request. When adding a new chain, register both the legacy alias and `CAIP2Network` in every `case` arm. diff --git a/cmd/obol/model.go b/cmd/obol/model.go index b9e97a5c..610bd957 100644 --- a/cmd/obol/model.go +++ b/cmd/obol/model.go @@ -465,6 +465,12 @@ func modelSetupCustomCommand(cfg *config.Config) *cli.Command { type modelStatusResult struct { Providers []modelStatusProvider `json:"providers"` Discovered []discoverProvider `json:"discovered,omitempty"` + // RouterDrift is set when the live LiteLLM router disagrees with the + // litellm-config ConfigMap (a hot-add/hot-delete failed or never ran). + // Omitted when the check passed; RouterDriftError explains when the + // check itself could not run (cluster or pod unavailable). + RouterDrift *model.RouterDrift `json:"router_drift,omitempty"` + RouterDriftError string `json:"router_drift_error,omitempty"` } type modelStatusProvider struct { @@ -498,10 +504,21 @@ func modelStatusCommand(cfg *config.Config) *cli.Command { defer cancel() discovered, _ := model.DiscoverLocalProviders(scanCtx) + // Drift safety net: Reloader no longer restarts LiteLLM on + // litellm-config changes, so a silently-failed hot-add/hot-delete + // is only visible here. Non-fatal — status must still render + // when the cluster or pod is unreachable. + drift, driftErr := model.CheckRouterDrift(cfg) + if u.IsJSON() { result := modelStatusResult{ Discovered: discoveredProvidersToJSON(discovered), } + if driftErr != nil { + result.RouterDriftError = driftErr.Error() + } else if !drift.Empty() { + result.RouterDrift = &drift + } for _, name := range providers { s := status[name] key := "n/a" @@ -548,6 +565,21 @@ func modelStatusCommand(cfg *config.Config) *cli.Command { } u.Blank() + switch { + case driftErr != nil: + u.Dim(fmt.Sprintf(" Live router check unavailable: %v", driftErr)) + u.Blank() + case !drift.Empty(): + u.Warnf("LiteLLM live router disagrees with litellm-config (hot update failed?):") + for _, m := range drift.Missing { + u.Printf(" missing from router: %s (requests 404 until next LiteLLM rollout)", m) + } + for _, m := range drift.Extra { + u.Printf(" extra in router: %s (disappears on next LiteLLM rollout)", m) + } + u.Dim(" Fix: `obol kubectl rollout restart deployment/litellm -n llm` reloads the ConfigMap.") + u.Blank() + } if printModelRanking(u, cfg, "section") { u.Blank() } diff --git a/flows/buy-external.sh b/flows/buy-external.sh index e3d3f7d8..bd054edb 100755 --- a/flows/buy-external.sh +++ b/flows/buy-external.sh @@ -222,7 +222,7 @@ external_snapshot_on_fail() { python3 -c " import urllib.request, json try: - resp = urllib.request.urlopen('http://localhost:8402/status', timeout=5) + resp = urllib.request.urlopen('http://x402-buyer.llm.svc.cluster.local:8402/status', timeout=5) print(json.dumps(json.loads(resp.read()), indent=2)) except Exception as e: print(json.dumps({'error': repr(e)})) @@ -478,7 +478,7 @@ bob kubectl exec -n llm deployment/litellm -c litellm -- \ python3 -c " import urllib.request, json, sys try: - resp = urllib.request.urlopen('http://localhost:8402/status', timeout=5) + resp = urllib.request.urlopen('http://x402-buyer.llm.svc.cluster.local:8402/status', timeout=5) print(json.dumps(json.loads(resp.read()), indent=2)) except Exception as e: print(json.dumps({'error': repr(e)})) @@ -550,7 +550,7 @@ bob kubectl rollout status deployment/litellm -n llm --timeout=180s >/dev/null 2 sidecar_status_raw=$(bob kubectl exec -n llm deployment/litellm -c litellm -- \ python3 -c " import urllib.request -print(urllib.request.urlopen('http://localhost:8402/status', timeout=5).read().decode()) +print(urllib.request.urlopen('http://x402-buyer.llm.svc.cluster.local:8402/status', timeout=5).read().decode()) " 2>/dev/null || true) PAID_MODEL=$(printf '%s' "$sidecar_status_raw" | python3 -c " import json, sys @@ -682,7 +682,7 @@ bob kubectl exec -n llm deployment/litellm -c litellm -- \ python3 -c " import urllib.request, json try: - resp = urllib.request.urlopen('http://localhost:8402/status', timeout=5) + resp = urllib.request.urlopen('http://x402-buyer.llm.svc.cluster.local:8402/status', timeout=5) print(json.dumps(json.loads(resp.read()), indent=2)) except Exception as e: print(json.dumps({'error': repr(e)})) diff --git a/flows/flow-06-sell-setup.sh b/flows/flow-06-sell-setup.sh index 7cba7d61..e9ab454d 100755 --- a/flows/flow-06-sell-setup.sh +++ b/flows/flow-06-sell-setup.sh @@ -119,19 +119,21 @@ if [ "$litellm_port" = "4000" ]; then else fail "LiteLLM service port unexpected: $litellm_port (expected 4000)" fi -# Verify LiteLLM pod has 2 containers (litellm + x402-buyer sidecar) -step "LiteLLM pod has 2 containers (litellm + x402-buyer sidecar)" -container_count=$("$OBOL" kubectl get pods -n llm --no-headers 2>&1 | awk '{print $2}' | head -1) -if [ "$container_count" = "2/2" ]; then - pass "LiteLLM pod has 2/2 containers (litellm + x402-buyer sidecar)" +# x402-buyer runs as its own Deployment since the buyer split (litellm pod +# is single-container and stateless for zero-downtime rollouts). +step "x402-buyer deployment ready (standalone, no longer a litellm sidecar)" +buyer_ready=$("$OBOL" kubectl get deploy x402-buyer -n llm \ + -o jsonpath='{.status.readyReplicas}' 2>&1) || true +if [ "$buyer_ready" = "1" ]; then + pass "x402-buyer deployment has 1 ready replica" else - fail "LiteLLM pod container count unexpected: $container_count (expected 2/2)" + fail "x402-buyer deployment not ready: readyReplicas=$buyer_ready (expected 1)" fi -# Verify x402-buyer sidecar health (serves /healthz at port 8402 in litellm pod) -step "x402-buyer sidecar healthy (buy-side payment handler)" +# Verify x402-buyer health (serves /healthz at port 8402 via its Service) +step "x402-buyer healthy (buy-side payment handler)" kill $(lsof -ti:8402) 2>/dev/null || true -"$OBOL" kubectl port-forward -n llm deployment/litellm 8402:8402 &>/dev/null & +"$OBOL" kubectl port-forward -n llm svc/x402-buyer 8402:8402 &>/dev/null & PF_BUYER_PID=$! for i in $(seq 1 8); do if curl -sf --max-time 2 http://localhost:8402/healthz >/dev/null 2>&1; then @@ -142,9 +144,9 @@ done buyer_health=$(curl -sf --max-time 5 http://localhost:8402/healthz 2>&1) || true cleanup_pid "$PF_BUYER_PID" if echo "$buyer_health" | grep -q "ok"; then - pass "x402-buyer sidecar healthy: $buyer_health" + pass "x402-buyer healthy: $buyer_health" else - fail "x402-buyer sidecar health check failed — ${buyer_health:0:100}" + fail "x402-buyer health check failed — ${buyer_health:0:100}" fi if [ "$SELL_UPSTREAM_SERVICE" = "ollama" ]; then run_step_grep "Ollama reachable" "models" curl -sf http://localhost:11434/api/tags diff --git a/flows/flow-08-buy.sh b/flows/flow-08-buy.sh index eed7c82d..1de2a63f 100755 --- a/flows/flow-08-buy.sh +++ b/flows/flow-08-buy.sh @@ -68,7 +68,7 @@ buyer_sidecar_status() { python3 -c " import urllib.request, json try: - resp = urllib.request.urlopen('http://localhost:8402/status', timeout=5) + resp = urllib.request.urlopen('http://x402-buyer.llm.svc.cluster.local:8402/status', timeout=5) d = json.loads(resp.read()) for name, info in d.items(): print('%s: remaining=%d spent=%d model=%s' % (name, info['remaining'], info['spent'], info['public_model'])) diff --git a/flows/flow-11-dual-stack.sh b/flows/flow-11-dual-stack.sh index 19b61c2b..fe092760 100755 --- a/flows/flow-11-dual-stack.sh +++ b/flows/flow-11-dual-stack.sh @@ -308,7 +308,7 @@ buyer_sidecar_status() { python3 -c " import urllib.request, json try: - resp = urllib.request.urlopen('http://localhost:8402/status', timeout=5) + resp = urllib.request.urlopen('http://x402-buyer.llm.svc.cluster.local:8402/status', timeout=5) d = json.loads(resp.read()) for name, info in d.items(): print('%s: remaining=%d spent=%d model=%s' % (name, info['remaining'], info['spent'], info['public_model'])) diff --git a/flows/lib-dual-stack.sh b/flows/lib-dual-stack.sh index b949f37e..b8a72d76 100644 --- a/flows/lib-dual-stack.sh +++ b/flows/lib-dual-stack.sh @@ -400,7 +400,7 @@ buyer_sidecar_status() { python3 -c " import urllib.request, json try: - resp = urllib.request.urlopen('http://localhost:8402/status', timeout=5) + resp = urllib.request.urlopen('http://x402-buyer.llm.svc.cluster.local:8402/status', timeout=5) d = json.loads(resp.read()) for name, info in d.items(): print('%s: remaining=%d spent=%d model=%s' % (name, info['remaining'], info['spent'], info['public_model'])) diff --git a/internal/embed/embed_buyer_state_test.go b/internal/embed/embed_buyer_state_test.go index db99ef81..38b841ed 100644 --- a/internal/embed/embed_buyer_state_test.go +++ b/internal/embed/embed_buyer_state_test.go @@ -3,13 +3,18 @@ package embed import "testing" // TestBuyerStatePVC asserts that x402-buyer's /state is backed by a PVC -// (not an emptyDir), and that the litellm Deployment uses the Recreate -// strategy so the RWO PVC can be remounted without overlap. +// (not an emptyDir), that the buyer runs as its own Recreate Deployment so +// the RWO PVC is remounted without overlap, and that the litellm Deployment +// is stateless with RollingUpdate maxUnavailable: 0 (zero-downtime rollouts, +// issue #321). // -// Regression: emptyDir lost consumed.json on every pod restart, causing +// Regression (emptyDir): lost consumed.json on every pod restart, causing // the buyer to re-spend already-consumed auths from the ConfigMap pool // and cascading into facilitator 400s ("nonce already used") until a // manual `buy.py process --all` reseeded. +// Regression (sidecar coupling): while the buyer lived in the litellm pod, +// the RWO PVC forced litellm onto Recreate — every rollout (Reloader secret +// change, image bump) was a full inference gap. func TestBuyerStatePVC(t *testing.T) { data, err := ReadInfrastructureFile("base/templates/llm.yaml") if err != nil { @@ -41,15 +46,15 @@ func TestBuyerStatePVC(t *testing.T) { t.Error("PVC missing spec.resources.requests.storage") } - // litellm Deployment volume entry must reference the PVC, not emptyDir. - dep := findDocByName(docs, "Deployment", "litellm") - if dep == nil { - t.Fatal("litellm Deployment missing from llm.yaml") + // The x402-buyer Deployment owns the PVC mount; litellm must not. + buyerDep := findDocByName(docs, "Deployment", "x402-buyer") + if buyerDep == nil { + t.Fatal("x402-buyer Deployment missing from llm.yaml (buyer split, issue #321)") } - volumes, ok := nested(dep, "spec", "template", "spec", "volumes").([]any) + volumes, ok := nested(buyerDep, "spec", "template", "spec", "volumes").([]any) if !ok { - t.Fatal("litellm Deployment has no volumes") + t.Fatal("x402-buyer Deployment has no volumes") } var stateVolume map[string]any @@ -65,7 +70,7 @@ func TestBuyerStatePVC(t *testing.T) { } if stateVolume == nil { - t.Fatal("litellm Deployment missing 'x402-buyer-state' volume") + t.Fatal("x402-buyer Deployment missing 'x402-buyer-state' volume") } if _, isEmptyDir := stateVolume["emptyDir"]; isEmptyDir { @@ -81,45 +86,67 @@ func TestBuyerStatePVC(t *testing.T) { t.Errorf("persistentVolumeClaim.claimName = %v, want x402-buyer-state", claim) } - // Strategy must be Recreate so the new pod waits for the old pod to - // release the RWO PVC before mounting. RollingUpdate with maxSurge>0 - // would block indefinitely. - if strat := nested(dep, "spec", "strategy", "type"); strat != "Recreate" { - t.Errorf("litellm Deployment strategy.type = %v, want Recreate (RWO PVC cannot be co-mounted during surge)", strat) + // Buyer strategy must be Recreate: the new pod waits for the old pod to + // release the RWO PVC before mounting, and consumed-auth state must + // have exactly one writer (double-spend protection). + if strat := nested(buyerDep, "spec", "strategy", "type"); strat != "Recreate" { + t.Errorf("x402-buyer Deployment strategy.type = %v, want Recreate (RWO PVC + single-writer auth state)", strat) } - if policy := nested(dep, "spec", "template", "spec", "securityContext", "fsGroupChangePolicy"); policy != "OnRootMismatch" { - t.Errorf("litellm pod fsGroupChangePolicy = %v, want OnRootMismatch", policy) + if policy := nested(buyerDep, "spec", "template", "spec", "securityContext", "fsGroupChangePolicy"); policy != "OnRootMismatch" { + t.Errorf("x402-buyer pod fsGroupChangePolicy = %v, want OnRootMismatch", policy) } - // x402-buyer must keep container-level UID/GID 1000 while hostPath PVs - // from <= v0.10.0-rc12 clusters remain in support: those PVs ignore the - // pod fsGroup (kubelet skips ownership management on hostPath) and hold - // a consumed.json written 0600 by UID 1000 — a 65532 sidecar crashloops + // The buyer pod must keep UID/GID 1000 while hostPath PVs from + // <= v0.10.0-rc12 clusters remain in support: those PVs ignore fsGroup + // (kubelet skips ownership management on hostPath) and hold a + // consumed.json written 0600 by UID 1000 — a 65532 buyer crashloops // on `load state` and takes every paid/* route down. On fresh local-type - // PVs the explicit UID is harmless (fsGroup 65532 grants group access). + // PVs the explicit UID is harmless. + if u := nested(buyerDep, "spec", "template", "spec", "securityContext", "runAsUser"); u != 1000 { + t.Errorf("x402-buyer pod securityContext.runAsUser = %v, want 1000 (legacy hostPath-PV state compat)", u) + } + if g := nested(buyerDep, "spec", "template", "spec", "securityContext", "runAsGroup"); g != 1000 { + t.Errorf("x402-buyer pod securityContext.runAsGroup = %v, want 1000 (legacy hostPath-PV state compat)", g) + } + if nr := nested(buyerDep, "spec", "template", "spec", "securityContext", "runAsNonRoot"); nr != true { + t.Errorf("x402-buyer pod securityContext.runAsNonRoot = %v, want true (restricted PSS)", nr) + } + + // litellm must be stateless: no buyer container, no PVC volume, and + // RollingUpdate with maxUnavailable: 0 so rollouts never gap inference. + dep := findDocByName(docs, "Deployment", "litellm") + if dep == nil { + t.Fatal("litellm Deployment missing from llm.yaml") + } + if liteVols, ok := nested(dep, "spec", "template", "spec", "volumes").([]any); ok { + for _, v := range liteVols { + vm, ok := v.(map[string]any) + if !ok { + continue + } + if _, isPVC := vm["persistentVolumeClaim"]; isPVC { + t.Errorf("litellm Deployment mounts PVC volume %v — litellm must stay stateless for RollingUpdate maxUnavailable:0", vm["name"]) + } + } + } containers, ok := nested(dep, "spec", "template", "spec", "containers").([]any) if !ok { t.Fatal("litellm Deployment has no containers") } - var buyer map[string]any for _, c := range containers { cm, ok := c.(map[string]any) if ok && cm["name"] == "x402-buyer" { - buyer = cm - break + t.Error("x402-buyer container found in litellm Deployment — the buyer must run as its own Deployment (issue #321)") } } - if buyer == nil { - t.Fatal("x402-buyer container missing from litellm Deployment") - } - if u := nested(buyer, "securityContext", "runAsUser"); u != 1000 { - t.Errorf("x402-buyer securityContext.runAsUser = %v, want 1000 (legacy hostPath-PV state compat)", u) + if strat := nested(dep, "spec", "strategy", "type"); strat != "RollingUpdate" { + t.Errorf("litellm Deployment strategy.type = %v, want RollingUpdate (zero-downtime rollouts)", strat) } - if g := nested(buyer, "securityContext", "runAsGroup"); g != 1000 { - t.Errorf("x402-buyer securityContext.runAsGroup = %v, want 1000 (legacy hostPath-PV state compat)", g) + if mu := nested(dep, "spec", "strategy", "rollingUpdate", "maxUnavailable"); mu != 0 { + t.Errorf("litellm Deployment rollingUpdate.maxUnavailable = %v, want 0 (a new pod must be Ready before the old one terminates)", mu) } - if nr := nested(buyer, "securityContext", "runAsNonRoot"); nr != true { - t.Errorf("x402-buyer securityContext.runAsNonRoot = %v, want true (restricted PSS)", nr) + if ms := nested(dep, "spec", "strategy", "rollingUpdate", "maxSurge"); ms != 1 { + t.Errorf("litellm Deployment rollingUpdate.maxSurge = %v, want 1", ms) } } diff --git a/internal/embed/infrastructure/base/templates/llm.yaml b/internal/embed/infrastructure/base/templates/llm.yaml index 917d2bfd..cd50bdf2 100644 --- a/internal/embed/infrastructure/base/templates/llm.yaml +++ b/internal/embed/infrastructure/base/templates/llm.yaml @@ -23,7 +23,7 @@ metadata: name: llm labels: # Pod Security Standards: Restricted profile enforced at admission. - # The litellm pod (litellm + x402-buyer sidecar) runs as non-root with + # The litellm and x402-buyer pods run as non-root with # all caps dropped, seccomp=RuntimeDefault, and readOnlyRootFilesystem; # write paths are routed to named emptyDir mounts. pod-security.kubernetes.io/enforce: restricted @@ -82,7 +82,11 @@ data: - model_name: "paid/*" litellm_params: model: "openai/*" - api_base: "http://127.0.0.1:8402/v1" + # x402-buyer runs as its own Deployment/Service (not a sidecar) so + # LiteLLM stays stateless and can roll with maxUnavailable: 0. + # The /v1 suffix is required — LiteLLM's OpenAI provider does not + # append it (CLAUDE.md pitfall 6). + api_base: "http://x402-buyer.llm.svc.cluster.local:8402/v1" api_key: "unused" general_settings: master_key: os.environ/LITELLM_MASTER_KEY @@ -133,10 +137,6 @@ stringData: # facilitator's nonce protection until a manual buy.py process --all. # PVC backed by local-path (single-node k3d default storage class) # gives crash-safety without conversion to StatefulSet. -# -# Deployment strategy: Recreate — RWO PVC can't be mounted by two -# pods, so RollingUpdate's surge would block. Recreate accepts a -# brief gap during rollout (litellm is replicas:1 anyway). apiVersion: v1 kind: PersistentVolumeClaim metadata: @@ -158,16 +158,23 @@ metadata: labels: app: litellm spec: - # Keep a single LiteLLM + x402-buyer pod while buyer auth consumption state - # is local to the sidecar pod. Scale this back out only after consumed auth - # state is shared or auth pools are sharded per replica. + # Single replica is enough locally: with RollingUpdate maxUnavailable: 0 + # below, every rollout (image bump, secret rotation via Reloader) surges a + # new pod to Ready before the old one terminates — no inference gap. + # LiteLLM is stateless here (config is copied from the ConfigMap at pod + # start; x402-buyer runs as its own Deployment with the stateful PVC), so + # scaling out is safe if ever needed — but note the hot model APIs must + # then be fanned out per pod (the CLI already does; see litellmAPICall). replicas: 1 - # Recreate (not RollingUpdate) because the x402-buyer-state PVC is RWO and - # cannot be co-mounted by an overlapping new pod during surge. Litellm is - # replicas: 1 so this just trades the (currently maxSurge:1) overlap for a - # short gap during rollout — acceptable, and unavoidable with RWO storage. + # RollingUpdate with zero unavailability is the whole point of keeping + # this pod stateless (issue #321). The old Recreate strategy was forced by + # the buyer sidecar's RWO PVC, which now lives in the x402-buyer + # Deployment below. strategy: - type: Recreate + type: RollingUpdate + rollingUpdate: + maxSurge: 1 + maxUnavailable: 0 selector: matchLabels: app: litellm @@ -176,15 +183,23 @@ spec: labels: app: litellm annotations: - # litellm-config only: model-list changes must roll the pod so paid - # routes pick up new entries. The buyer ConfigMaps are deliberately - # NOT listed — x402-buyer hot-reloads them via /admin/reload (driven - # by the serviceoffer-controller on every buy/refill), and this - # Deployment is strategy Recreate with one replica: reloading on - # x402-buyer-config/x402-buyer-auths would take the whole inference - # gateway down on every purchase event (CLAUDE.md pitfall 7: restart - # is the fallback, not the default buy path). - configmap.reloader.stakater.com/reload: "litellm-config" + # litellm-secrets ONLY. Deliberately narrow: + # - litellm-config is NOT watched: every model_list change is + # hot-applied to the live router via /model/new + /model/delete by + # both the CLI (internal/model/model.go hotAddModels/hotDeleteModel) + # and the serviceoffer-controller (purchase_helpers.go). The + # ConfigMap is persistence for the next pod start, nothing more. + # Watching it here would roll the pod on every model add/remove/ + # prefer and on every first-time purchase — reintroducing the + # inference gap the hot APIs exist to remove (issue #321). + # - x402-buyer-config/x402-buyer-auths are NOT watched: x402-buyer + # hot-reloads them via /admin/reload driven by the controller on + # every buy/refill (CLAUDE.md pitfall 7: restart is the fallback, + # not the default buy path). + # - litellm-secrets IS watched: os.environ/ API-key references are + # resolved once at config load, so key changes genuinely need a + # pod replacement. With RollingUpdate maxUnavailable:0 below this + # is a gapless surge, not an outage. secret.reloader.stakater.com/reload: "litellm-secrets" spec: terminationGracePeriodSeconds: 60 @@ -300,9 +315,81 @@ spec: limits: cpu: "1" memory: 2Gi + volumes: + - name: litellm-config-source + configMap: + name: litellm-config + items: + - key: config.yaml + path: config.yaml + # Runtime copy of litellm-config. ConfigMap volumes are read-only, but + # LiteLLM's config-backed /model/new path persists to config.yaml after + # updating the live router. Keep the source ConfigMap immutable and give + # LiteLLM a pod-local writable working file instead. + - name: litellm-config-work + emptyDir: + sizeLimit: 1Mi + # Writable /tmp for Python tempfile / multipart uploads. Sized + # modestly — LiteLLM streams responses rather than buffering them. + - name: litellm-tmp + emptyDir: + sizeLimit: 128Mi + # Writable HOME for LiteLLM's pip/HF/XDG cache lookups so the + # container can run with readOnlyRootFilesystem=true. + - name: litellm-home + emptyDir: + sizeLimit: 256Mi + +--- +# x402-buyer as its own Deployment (was: sidecar in the litellm pod). +# Separating it makes LiteLLM stateless so LiteLLM can use RollingUpdate +# maxUnavailable: 0 (zero-downtime rollouts on image bumps and secret +# rotation, issue #321). The buyer keeps the RWO PVC and stays single +# replica + Recreate: consumed-auth state must have exactly one writer or +# pre-signed auths could be double-spent. Buyer rollouts briefly gap paid/* +# routes only — rare (buyer image bumps; buyer CM changes hot-reload via +# /admin/reload, driven by the serviceoffer-controller). +apiVersion: apps/v1 +kind: Deployment +metadata: + name: x402-buyer + namespace: llm + labels: + app: x402-buyer +spec: + replicas: 1 + strategy: + type: Recreate + selector: + matchLabels: + app: x402-buyer + template: + metadata: + labels: + app: x402-buyer + # No Reloader annotations: x402-buyer-config/x402-buyer-auths changes + # are hot-reloaded via /admin/reload (CLAUDE.md pitfall 7). + spec: + # PSS Restricted. UID/GID 1000 (not the 65532 distroless default): on + # fresh clusters the StorageClass provisions local PVs where kubelet + # applies fsGroup and any UID works, but clusters UPGRADED in place + # from <= v0.10.0-rc12 have a hostPath-typed x402-buyer-state PV + # (kubelet ignores fsGroup there) chowned 1000:1000 with consumed.json + # written 0600 by UID 1000 — a 65532 buyer cannot read it and + # crashloops on startup (`load state` Fatalf), taking every paid/* + # route down. Keep 1000 until hostPath PVs are out of support. + securityContext: + runAsNonRoot: true + runAsUser: 1000 + runAsGroup: 1000 + fsGroup: 1000 + fsGroupChangePolicy: OnRootMismatch + seccompProfile: + type: RuntimeDefault + containers: - name: x402-buyer # Pinned by sha256 digest (multi-arch manifest list, amd64+arm64) - # so the deployed sidecar is byte-for-byte identical across QA + # so the deployed buyer is byte-for-byte identical across QA # hosts. The :62f11c9 tag is preserved for human readability; the # digest is authoritative. # Previous tag-only pin allowed the local-build path to silently @@ -312,20 +399,7 @@ spec: # across flow-08/11/14/13. See internal/embed/embed_image_pin_test.go. image: ghcr.io/obolnetwork/x402-buyer:62f11c9@sha256:c58f7a20303538311d7e207e816e699c038bd8ded6339697fc0a01ed187abcda imagePullPolicy: IfNotPresent - # PSS Restricted + writable PVC. On fresh clusters the StorageClass - # asks local-path-provisioner for local PVs, so kubelet applies the - # pod-level fsGroup to /state on mount and any UID works. The - # explicit UID/GID 1000 below is for clusters UPGRADED in place - # from <= v0.10.0-rc12: their x402-buyer-state PV is hostPath-typed - # (kubelet ignores fsGroup there) and the old provisioner chowned - # the dir 1000:1000 with consumed.json written 0600 by UID 1000 — - # a 65532 sidecar cannot even read it and crashloops on startup - # (`load state` Fatalf), taking every paid/* route down. Keep 1000 - # until hostPath PVs are out of support. securityContext: - runAsNonRoot: true - runAsUser: 1000 - runAsGroup: 1000 allowPrivilegeEscalation: false readOnlyRootFilesystem: true capabilities: @@ -368,29 +442,6 @@ spec: - name: x402-buyer-state mountPath: /state volumes: - - name: litellm-config-source - configMap: - name: litellm-config - items: - - key: config.yaml - path: config.yaml - # Runtime copy of litellm-config. ConfigMap volumes are read-only, but - # LiteLLM's config-backed /model/new path persists to config.yaml after - # updating the live router. Keep the source ConfigMap immutable and give - # LiteLLM a pod-local writable working file instead. - - name: litellm-config-work - emptyDir: - sizeLimit: 1Mi - # Writable /tmp for Python tempfile / multipart uploads. Sized - # modestly — LiteLLM streams responses rather than buffering them. - - name: litellm-tmp - emptyDir: - sizeLimit: 128Mi - # Writable HOME for LiteLLM's pip/HF/XDG cache lookups so the - # container can run with readOnlyRootFilesystem=true. - - name: litellm-home - emptyDir: - sizeLimit: 256Mi - name: buyer-config configMap: name: x402-buyer-config @@ -403,6 +454,30 @@ spec: persistentVolumeClaim: claimName: x402-buyer-state +--- +# Stable in-cluster address for the buyer: LiteLLM's paid/* routes point +# api_base at this Service, and agents' buy.py reaches /status through it. +# Exposure note: pod IP :8402 was already cluster-reachable when the buyer +# was a sidecar (no NetworkPolicy in this namespace); the Service changes +# addressing, not the security posture. Auth spend is bounded by the +# pre-signed pool (max loss = N x price). +apiVersion: v1 +kind: Service +metadata: + name: x402-buyer + namespace: llm + labels: + app: x402-buyer +spec: + type: ClusterIP + selector: + app: x402-buyer + ports: + - name: buyer-http + port: 8402 + targetPort: buyer-http + protocol: TCP + --- apiVersion: policy/v1 kind: PodDisruptionBudget @@ -435,15 +510,16 @@ spec: --- # Relocated from helmfile.yaml `llm-buyer-podmonitor` bedag/raw release. -# Lives alongside its workload (litellm + x402-buyer sidecar) instead of +# Lives alongside its workload (the x402-buyer Deployment) instead of # inlined in helmfile so the chart layout is the single source of truth # for what ships in the llm namespace. The PodMonitor CRD comes from the # monitoring release (kube-prometheus-stack), so `base` now declares a # `needs: [monitoring/monitoring]` in helmfile.yaml to guarantee CRD # presence before this template applies. # -# PodMonitor (not ServiceMonitor) because the sidecar listens on a per-pod -# port (8402) that is NOT exposed via the litellm Service. +# PodMonitor kept under its historical name (litellm-x402-buyer) so +# helmfile ordering notes and tests stay stable; it now targets the +# standalone x402-buyer pods (the buyer is no longer a litellm sidecar). # Picked up by kube-prometheus-stack via the `release: monitoring` label. apiVersion: monitoring.coreos.com/v1 kind: PodMonitor @@ -452,11 +528,11 @@ metadata: namespace: llm labels: release: monitoring - app: litellm + app: x402-buyer spec: selector: matchLabels: - app: litellm + app: x402-buyer podMetricsEndpoints: - port: buyer-http path: /metrics diff --git a/internal/embed/skills/buy-x402/SKILL.md b/internal/embed/skills/buy-x402/SKILL.md index 71a9e8de..cda04167 100644 --- a/internal/embed/skills/buy-x402/SKILL.md +++ b/internal/embed/skills/buy-x402/SKILL.md @@ -357,9 +357,9 @@ flowchart LR 4. **Reconcile**: The controller validates pricing, writes per-upstream buyer config/auth files into the `x402-buyer-config` and `x402-buyer-auths` ConfigMaps in `llm`, and keeps the paid model route available in LiteLLM. -5. **Runtime mount**: A lean Go sidecar (`x402-buyer`) already runs inside the existing `litellm` pod in the `llm` namespace. It mounts both ConfigMaps and serves as an OpenAI-compatible reverse proxy on `127.0.0.1:8402`. +5. **Runtime mount**: A lean Go service (`x402-buyer`) runs as its own Deployment in the `llm` namespace. It mounts both ConfigMaps and serves as an OpenAI-compatible reverse proxy at `x402-buyer.llm.svc.cluster.local:8402`. -6. **Wire**: LiteLLM keeps one static wildcard route: `paid/* -> openai/* -> 127.0.0.1:8402/v1`. The controller also adds explicit paid-model entries when required so models with colons resolve reliably. The public model name is always `paid/`. +6. **Wire**: LiteLLM keeps one static wildcard route: `paid/* -> openai/* -> http://x402-buyer.llm.svc.cluster.local:8402/v1`. The controller also adds explicit paid-model entries when required so models with colons resolve reliably. The public model name is always `paid/`. 7. **Runtime**: On each request through the sidecar: - Sidecar forwards to upstream seller diff --git a/internal/embed/skills/buy-x402/references/x402-buyer-api.md b/internal/embed/skills/buy-x402/references/x402-buyer-api.md index 2f705ad5..37bec510 100644 --- a/internal/embed/skills/buy-x402/references/x402-buyer-api.md +++ b/internal/embed/skills/buy-x402/references/x402-buyer-api.md @@ -228,12 +228,12 @@ model_list: - model_name: "paid/*" litellm_params: model: "openai/*" - api_base: "http://127.0.0.1:8402/v1" + api_base: "http://x402-buyer.llm.svc.cluster.local:8402/v1" api_key: "unused" - model_name: "paid/qwen3.5:9b" litellm_params: model: "openai/paid/qwen3.5:9b" - api_base: "http://127.0.0.1:8402/v1" + api_base: "http://x402-buyer.llm.svc.cluster.local:8402/v1" api_key: "unused" ``` diff --git a/internal/embed/skills/buy-x402/scripts/buy.py b/internal/embed/skills/buy-x402/scripts/buy.py index 8b0b571f..dc1d6cf0 100644 --- a/internal/embed/skills/buy-x402/scripts/buy.py +++ b/internal/embed/skills/buy-x402/scripts/buy.py @@ -62,7 +62,10 @@ DEFAULT_CHAIN = os.environ.get("ERPC_NETWORK", "base-sepolia") BUYER_NS = "llm" -LITELLM_DEPLOY = "litellm" +# x402-buyer runs as its own Deployment (label app=x402-buyer) since the +# litellm pod went stateless for zero-downtime rollouts; it is no longer a +# sidecar in the litellm pod. +BUYER_DEPLOY = "x402-buyer" BUYER_PORT = 8402 # Some sellers sit behind a Cloudflare WAF that blocks the default @@ -335,13 +338,13 @@ def _parse_money_amount(value, asset, extra=None): # --------------------------------------------------------------------------- -# Buyer sidecar status helpers +# Buyer status helpers # --------------------------------------------------------------------------- -def _get_litellm_pod(token, ssl_ctx): - """Return the current LiteLLM pod object, or None if unavailable.""" +def _get_buyer_pod(token, ssl_ctx): + """Return the current x402-buyer pod object, or None if unavailable.""" pods = api_get( - f"/api/v1/namespaces/{BUYER_NS}/pods?labelSelector=app={LITELLM_DEPLOY}", + f"/api/v1/namespaces/{BUYER_NS}/pods?labelSelector=app={BUYER_DEPLOY}", token, ssl_ctx, ) for item in pods.get("items", []): @@ -353,10 +356,10 @@ def _get_litellm_pod(token, ssl_ctx): def _buyer_status(): - """Return live sidecar status JSON, or None if the sidecar is unavailable.""" + """Return live buyer status JSON, or None if the buyer is unavailable.""" token, _ = load_sa() ssl_ctx = make_ssl_context() - pod = _get_litellm_pod(token, ssl_ctx) + pod = _get_buyer_pod(token, ssl_ctx) if not pod: return None @@ -2344,11 +2347,11 @@ def cmd_status(name): print(f"Auths spent: {live_status.get('spent', status.get('spent', 0))}") print() - pod = _get_litellm_pod(token, ssl_ctx) + pod = _get_buyer_pod(token, ssl_ctx) if not pod: - print("Sidecar: NOT RUNNING (LiteLLM pod unavailable)") + print("Buyer: NOT RUNNING (x402-buyer pod unavailable)") else: - print(f"Sidecar: {pod.get('status', {}).get('phase', 'Unknown')}") + print(f"Buyer: {pod.get('status', {}).get('phase', 'Unknown')}") # --------------------------------------------------------------------------- diff --git a/internal/model/drift.go b/internal/model/drift.go new file mode 100644 index 00000000..a063df26 --- /dev/null +++ b/internal/model/drift.go @@ -0,0 +1,147 @@ +package model + +import ( + "encoding/json" + "errors" + "fmt" + "os" + "path/filepath" + "strings" + + "github.com/ObolNetwork/obol-stack/internal/config" + "github.com/ObolNetwork/obol-stack/internal/kubectl" + "gopkg.in/yaml.v3" +) + +// RouterDrift describes a divergence between the litellm-config ConfigMap +// (the persistence source of truth) and the live LiteLLM router. +// +// Since Reloader no longer watches litellm-config (issue #321: a +// ConfigMap-triggered rollout would gap inference on every model change), +// hot-add/hot-delete API calls are the only thing keeping the live router in +// sync between pod restarts. This check is the replacement safety net: it +// makes a silently-failed hot call visible in `obol model status` instead of +// leaving the operator with a router that quietly disagrees with the config. +type RouterDrift struct { + // Missing entries exist in the ConfigMap model_list but not in the live + // router — a hot-add failed or never ran. They will appear after the next + // pod restart; until then requests to them 404. + Missing []string `json:"missing,omitempty"` + // Extra entries are served by the live router but absent from the + // ConfigMap — a hot-delete failed or never ran. They disappear on the + // next pod restart. + Extra []string `json:"extra,omitempty"` +} + +// Empty reports whether the live router and the ConfigMap agree. +func (d RouterDrift) Empty() bool { + return len(d.Missing) == 0 && len(d.Extra) == 0 +} + +// DiffRouterModels is the pure core of the drift check. configured is the +// raw model_name list from the ConfigMap; live is the model id list reported +// by the running router (/v1/models). +// +// Wildcard entries (trailing "/*") are configuration for LiteLLM's router, +// not concrete serving targets: a configured wildcard is never "missing", +// and any live model matched by a configured wildcard prefix is not "extra". +func DiffRouterModels(configured, live []string) RouterDrift { + liveSet := make(map[string]struct{}, len(live)) + for _, m := range live { + liveSet[m] = struct{}{} + } + + var wildcardPrefixes []string + configuredSet := make(map[string]struct{}, len(configured)) + var drift RouterDrift + + for _, name := range configured { + if prefix, ok := strings.CutSuffix(name, "*"); ok { + wildcardPrefixes = append(wildcardPrefixes, prefix) + configuredSet[name] = struct{}{} + continue + } + configuredSet[name] = struct{}{} + if _, ok := liveSet[name]; !ok { + drift.Missing = append(drift.Missing, name) + } + } + + for _, name := range live { + if _, ok := configuredSet[name]; ok { + continue + } + if strings.HasSuffix(name, "*") { + // The live router lists wildcard groups verbatim; they are + // router config, not drift. + continue + } + matched := false + for _, prefix := range wildcardPrefixes { + if strings.HasPrefix(name, prefix) { + matched = true + break + } + } + if !matched { + drift.Extra = append(drift.Extra, name) + } + } + + return drift +} + +// CheckRouterDrift compares the litellm-config ConfigMap model_list against +// the live router's /v1/models. It returns an error when either side cannot +// be read (cluster down, pod not running) — callers should treat that as +// "check unavailable", not as drift. +func CheckRouterDrift(cfg *config.Config) (RouterDrift, error) { + kubectlBinary := filepath.Join(cfg.BinDir, "kubectl") + kubeconfigPath := filepath.Join(cfg.ConfigDir, "kubeconfig.yaml") + + if _, err := os.Stat(kubeconfigPath); os.IsNotExist(err) { + return RouterDrift{}, errors.New("cluster not running") + } + + raw, err := kubectl.Output(kubectlBinary, kubeconfigPath, + "get", "configmap", configMapName, "-n", namespace, "-o", "jsonpath={.data.config\\.yaml}") + if err != nil { + return RouterDrift{}, fmt.Errorf("read LiteLLM config: %w", err) + } + + var litellmConfig LiteLLMConfig + if err := yaml.Unmarshal([]byte(raw), &litellmConfig); err != nil { + return RouterDrift{}, fmt.Errorf("parse LiteLLM config: %w", err) + } + + configured := make([]string, 0, len(litellmConfig.ModelList)) + for _, entry := range litellmConfig.ModelList { + configured = append(configured, entry.ModelName) + } + + masterKey, err := GetMasterKey(cfg) + if err != nil { + return RouterDrift{}, fmt.Errorf("read master key: %w", err) + } + + body, err := litellmGETViaPortForward(kubectlBinary, kubeconfigPath, masterKey, "/v1/models") + if err != nil { + return RouterDrift{}, fmt.Errorf("query live router: %w", err) + } + + var resp struct { + Data []struct { + ID string `json:"id"` + } `json:"data"` + } + if err := json.Unmarshal(body, &resp); err != nil { + return RouterDrift{}, fmt.Errorf("parse /v1/models response: %w", err) + } + + live := make([]string, 0, len(resp.Data)) + for _, m := range resp.Data { + live = append(live, m.ID) + } + + return DiffRouterModels(configured, live), nil +} diff --git a/internal/model/drift_test.go b/internal/model/drift_test.go new file mode 100644 index 00000000..aaeab1e7 --- /dev/null +++ b/internal/model/drift_test.go @@ -0,0 +1,89 @@ +package model + +import ( + "reflect" + "testing" +) + +// DiffRouterModels is the drift safety net that replaced the Reloader +// annotation on litellm-config (issue #321): with hot-add/hot-delete as the +// only live-apply path, a silently-failed hot call must surface as drift in +// `obol model status` instead of a router that quietly disagrees with the +// ConfigMap. +func TestDiffRouterModels(t *testing.T) { + tests := []struct { + name string + configured []string + live []string + wantMissing []string + wantExtra []string + }{ + { + name: "in sync", + configured: []string{"anthropic/claude-sonnet-5", "qwen3.5:9b"}, + live: []string{"anthropic/claude-sonnet-5", "qwen3.5:9b"}, + }, + { + name: "hot-add failed: configured model absent from router", + configured: []string{"qwen3.5:9b", "qwen3.5:4b"}, + live: []string{"qwen3.5:9b"}, + wantMissing: []string{"qwen3.5:4b"}, + }, + { + name: "hot-delete failed: removed model still served", + configured: []string{"qwen3.5:9b"}, + live: []string{"qwen3.5:9b", "qwen3.5:4b"}, + wantExtra: []string{"qwen3.5:4b"}, + }, + { + name: "configured wildcard is never missing", + configured: []string{"paid/*"}, + live: []string{}, + }, + { + name: "live model matched by configured wildcard is not extra", + configured: []string{"paid/*"}, + live: []string{"paid/qwen36-deep"}, + }, + { + name: "live wildcard group listed verbatim is not extra", + configured: []string{"paid/*"}, + live: []string{"paid/*"}, + }, + { + name: "wildcard does not cover other prefixes", + configured: []string{"paid/*", "qwen3.5:9b"}, + live: []string{"paid/qwen36-deep", "stale-model"}, + wantMissing: []string{"qwen3.5:9b"}, + wantExtra: []string{"stale-model"}, + }, + { + name: "both empty", + configured: nil, + live: nil, + }, + { + name: "empty config with live models", + live: []string{"orphan"}, + wantExtra: []string{"orphan"}, + }, + } + + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + got := DiffRouterModels(tt.configured, tt.live) + + if !reflect.DeepEqual(got.Missing, tt.wantMissing) { + t.Errorf("Missing = %v; want %v", got.Missing, tt.wantMissing) + } + if !reflect.DeepEqual(got.Extra, tt.wantExtra) { + t.Errorf("Extra = %v; want %v", got.Extra, tt.wantExtra) + } + + wantEmpty := len(tt.wantMissing) == 0 && len(tt.wantExtra) == 0 + if got.Empty() != wantEmpty { + t.Errorf("Empty() = %v; want %v", got.Empty(), wantEmpty) + } + }) + } +} diff --git a/internal/model/model.go b/internal/model/model.go index de2737ef..553c5347 100644 --- a/internal/model/model.go +++ b/internal/model/model.go @@ -466,6 +466,10 @@ func PatchLiteLLMEntries(cfg *config.Config, u *ui.UI, entries []ModelEntry) err } // RestartLiteLLM restarts the LiteLLM deployment and waits for rollout. +// A rollout that does not converge within the timeout is an error — callers +// must not report success while LiteLLM may be down or serving stale config +// (issue #321: the old warn-and-succeed behavior silently left LiteLLM +// broken after failed rollouts). func RestartLiteLLM(cfg *config.Config, u *ui.UI, provider string) error { kubectlBinary := filepath.Join(cfg.BinDir, "kubectl") kubeconfigPath := filepath.Join(cfg.ConfigDir, "kubeconfig.yaml") @@ -480,12 +484,10 @@ func RestartLiteLLM(cfg *config.Config, u *ui.UI, provider string) error { if err := kubectl.Run(kubectlBinary, kubeconfigPath, "rollout", "status", "deployment/"+deployName, "-n", namespace, "--timeout=90s"); err != nil { - u.Warnf("LiteLLM rollout not confirmed: %v", err) - u.Print("The deployment may still be rolling out.") - } else { - u.Successf("LiteLLM configured with %s provider", provider) + return fmt.Errorf("LiteLLM rollout not confirmed within 90s: %w (check `obol kubectl get pods -n %s`)", err, namespace) } + u.Successf("LiteLLM configured with %s provider", provider) return nil } @@ -838,10 +840,10 @@ func reorderModelList(entries []ModelEntry, names []string) ([]ModelEntry, bool, // Returns an error if any of the requested names is not present in the // current model_list — typos should be loud, not silent no-ops. // -// LiteLLM has no model_list reorder API, so after the ConfigMap patch this -// rolls the LiteLLM Deployment so the new order takes effect (the -// /v1/models listing follows model_list order, and hermes/openclaw read -// the ConfigMap directly via GetConfiguredModels for the agent primary). +// ConfigMap-only operation: hermes/openclaw read the agent primary from the +// ConfigMap via GetConfiguredModels, and LiteLLM's router does not use +// model_list order for routing, so no pod restart is needed. The live +// /v1/models listing keeps the old order until the next natural restart. func PreferModels(cfg *config.Config, u *ui.UI, names []string) error { if len(names) == 0 { return errors.New("at least one model name is required") @@ -892,15 +894,12 @@ func PreferModels(cfg *config.Config, u *ui.UI, names []string) error { return fmt.Errorf("failed to patch ConfigMap: %w", err) } - // LiteLLM has no reorder API; restart the deployment so the new order - // takes effect (mostly cosmetic for /v1/models listings — agent primary - // is read from the ConfigMap directly via GetConfiguredModels, which is - // already correct after the patch above). - if err := RestartLiteLLM(cfg, u, "prefer"); err != nil { - u.Warnf("LiteLLM rollout failed: %v", err) - u.Dim(" The ConfigMap is updated; agent will pick up the new primary on next sync.") - } - + // No LiteLLM restart: model_list order is an obol convention, not a + // LiteLLM routing input. The agent primary is read from the ConfigMap + // directly via GetConfiguredModels, which is already correct after the + // patch above. The live router's /v1/models listing keeps the old order + // until the next natural pod replacement — cosmetic only, and not worth + // an inference gap (issue #321). return nil } diff --git a/internal/model/record.go b/internal/model/record.go index 0f0b71f9..f9d1d360 100644 --- a/internal/model/record.go +++ b/internal/model/record.go @@ -130,9 +130,15 @@ func ReconcileRecorded(cfg *config.Config, u *ui.UI) { u.Warnf("LiteLLM restart after model reconcile failed: %v", err) } case configChanged: - // Reloader restarts LiteLLM on litellm-config changes since rc14; - // no manual rollout needed for a ConfigMap-only change. + // Reloader deliberately does NOT watch litellm-config (issue #321: + // a CM-triggered rollout would gap inference on every model change), + // so a ConfigMap-only reconcile must roll the pod explicitly for the + // live router to pick up the recorded model_list. With RollingUpdate + // maxUnavailable:0 on the litellm Deployment this is a gapless surge. u.Infof("Restored recorded model list (%d models)", len(state.ModelList)) + if err := RestartLiteLLM(cfg, u, "recorded model config"); err != nil { + u.Warnf("LiteLLM restart after model reconcile failed: %v", err) + } } } diff --git a/internal/serviceoffercontroller/controller.go b/internal/serviceoffercontroller/controller.go index f94f7c6b..68193dc0 100644 --- a/internal/serviceoffercontroller/controller.go +++ b/internal/serviceoffercontroller/controller.go @@ -248,6 +248,11 @@ func (c *Controller) Run(ctx context.Context, workers int) error { log.Printf("serviceoffer-controller: ensure default AgentIdentity: %v", err) } + // Heal paid-route entries written before the x402-buyer split (they + // point at the removed litellm-pod sidecar address). One-shot, + // idempotent, no-op on fresh clusters. + c.migrateLegacyBuyerAPIBases(ctx, "llm") + if workers < 1 { workers = 1 } diff --git a/internal/serviceoffercontroller/purchase_helpers.go b/internal/serviceoffercontroller/purchase_helpers.go index 7e8b0491..0bdf422e 100644 --- a/internal/serviceoffercontroller/purchase_helpers.go +++ b/internal/serviceoffercontroller/purchase_helpers.go @@ -59,9 +59,9 @@ func (c *Controller) getLiteLLMMasterKey(ctx context.Context, ns string) (string } // hotAddLiteLLMModel adds a model via the LiteLLM /model/new HTTP API. The -// in-memory router is updated without a pod restart, preserving the x402-buyer -// sidecar's consumed-auth state (which lives in a pod-local emptyDir and would -// be wiped by a rollout). +// in-memory router is updated without a pod restart, so paid routes appear +// with zero inference downtime (issue #321 — Reloader deliberately does not +// watch litellm-config, making this hot call the primary apply path). // // Returns an error if the API call fails; callers fall back to a pod restart // only as a last resort — see addLiteLLMModelEntry. @@ -275,14 +275,33 @@ func (c *Controller) removeBuyerUpstream(ctx context.Context, ns, name string) { } } +// buyerAPIBase returns the LiteLLM api_base for paid routes: the standalone +// x402-buyer Deployment's Service. The /v1 suffix is required — LiteLLM's +// OpenAI provider does not append it (CLAUDE.md pitfall 6). +func buyerAPIBase(ns string) string { + return fmt.Sprintf("http://x402-buyer.%s.svc.cluster.local:8402/v1", ns) +} + +// legacyBuyerAPIBase is the pre-split address from when x402-buyer ran as a +// sidecar in the litellm pod. Entries still carrying it are dead upstreams +// (nothing listens on the litellm pod's localhost:8402 anymore) and are +// migrated in place by addLiteLLMModelEntry and +// migrateLegacyBuyerAPIBases. +const legacyBuyerAPIBase = "http://127.0.0.1:8402/v1" + // addLiteLLMModelEntry adds a paid/ route to LiteLLM. Writes the // ConfigMap (persistence across restarts) and then hot-adds via /model/new -// (no pod restart — preserves the buyer sidecar's consumed-auth state). +// (no pod restart — paid routes appear with zero inference downtime). +// +// If the hot-add API call fails, we do NOT fall back to a pod restart. +// Instead we log and rely on the ConfigMap being reloaded on the next +// natural rollout (litellm rolls gaplessly via RollingUpdate +// maxUnavailable: 0, but a restart here would still churn in-flight +// requests for no reason). // -// If the hot-add API call fails, we do NOT fall back to a pod restart: that -// would wipe the sidecar's emptyDir /state/consumed.json and cause the -// facilitator to reject previously-spent auths as double-spends. Instead we -// log and rely on the ConfigMap being reloaded on the next natural restart. +// An existing entry that still points at the legacy sidecar address is +// migrated to the x402-buyer Service in place (ConfigMap + hot +// delete/re-add) — it was a dead upstream anyway. func (c *Controller) addLiteLLMModelEntry(ctx context.Context, ns, modelName string) { cm, err := c.kubeClient.CoreV1().ConfigMaps(ns).Get(ctx, "litellm-config", metav1.GetOptions{}) if err != nil { @@ -299,20 +318,43 @@ func (c *Controller) addLiteLLMModelEntry(ctx context.Context, ns, modelName str return } - for _, entry := range cfg.ModelList { - if entry.ModelName == modelName { - return - } - } - entry := model.ModelEntry{ ModelName: modelName, LiteLLMParams: model.LiteLLMParams{ Model: "openai/" + modelName, - APIBase: "http://127.0.0.1:8402/v1", + APIBase: buyerAPIBase(ns), APIKey: "unused", }, } + + for i := range cfg.ModelList { + if cfg.ModelList[i].ModelName != modelName { + continue + } + if cfg.ModelList[i].LiteLLMParams.APIBase != legacyBuyerAPIBase { + return + } + // Legacy sidecar address: rewrite in place, persist, hot-sync. + cfg.ModelList[i].LiteLLMParams.APIBase = buyerAPIBase(ns) + rendered, err := yaml.Marshal(&cfg) + if err != nil { + log.Printf("purchase: failed to serialize litellm-config: %v", err) + return + } + cm.Data["config.yaml"] = string(rendered) + if _, err := c.kubeClient.CoreV1().ConfigMaps(ns).Update(ctx, cm, metav1.UpdateOptions{FieldManager: litellmConfigFieldManager}); err != nil { + log.Printf("purchase: failed to update litellm-config: %v", err) + return + } + if err := c.hotDeleteLiteLLMModel(ctx, ns, modelName); err != nil { + log.Printf("purchase: migrate %s: hot-delete failed: %v", modelName, err) + } + if err := c.hotAddLiteLLMModel(ctx, ns, cfg.ModelList[i]); err != nil { + log.Printf("purchase: migrate %s: hot-add failed: %v; relying on ConfigMap reload", modelName, err) + } + return + } + cfg.ModelList = append(cfg.ModelList, entry) rendered, err := yaml.Marshal(&cfg) @@ -443,12 +485,72 @@ func (c *Controller) removeLiteLLMModelEntry(ctx context.Context, ns, modelName } } -// triggerBuyerReload sends POST /admin/reload to the x402-buyer sidecar -// on all running litellm pods. Best-effort — the sidecar reloads on its -// own 5-second timer anyway. +// migrateLegacyBuyerAPIBases rewrites model_list entries that still point at +// the pre-split buyer sidecar address (127.0.0.1:8402 inside the litellm +// pod) to the standalone x402-buyer Service. One-shot at controller startup: +// this is what heals clusters upgraded in place, whose litellm-config still +// carries paid/ entries written before the buyer split. Those entries +// are dead upstreams until migrated, so the delete/re-add window is not a +// regression. Best-effort: failures are logged and the ConfigMap rewrite +// alone still fixes the next LiteLLM rollout. +func (c *Controller) migrateLegacyBuyerAPIBases(ctx context.Context, ns string) { + cm, err := c.kubeClient.CoreV1().ConfigMaps(ns).Get(ctx, "litellm-config", metav1.GetOptions{}) + if err != nil { + if !apierrors.IsNotFound(err) { + log.Printf("buyer-migrate: read litellm-config: %v", err) + } + return + } + + var cfg model.LiteLLMConfig + if err := yaml.Unmarshal([]byte(cm.Data["config.yaml"]), &cfg); err != nil { + log.Printf("buyer-migrate: parse litellm-config: %v", err) + return + } + + var migrated []model.ModelEntry + for i := range cfg.ModelList { + if cfg.ModelList[i].LiteLLMParams.APIBase != legacyBuyerAPIBase { + continue + } + cfg.ModelList[i].LiteLLMParams.APIBase = buyerAPIBase(ns) + migrated = append(migrated, cfg.ModelList[i]) + } + if len(migrated) == 0 { + return + } + + rendered, err := yaml.Marshal(&cfg) + if err != nil { + log.Printf("buyer-migrate: serialize litellm-config: %v", err) + return + } + if cm.Data == nil { + cm.Data = make(map[string]string) + } + cm.Data["config.yaml"] = string(rendered) + if _, err := c.kubeClient.CoreV1().ConfigMaps(ns).Update(ctx, cm, metav1.UpdateOptions{FieldManager: litellmConfigFieldManager}); err != nil { + log.Printf("buyer-migrate: update litellm-config: %v", err) + return + } + + for _, entry := range migrated { + if err := c.hotDeleteLiteLLMModel(ctx, ns, entry.ModelName); err != nil { + log.Printf("buyer-migrate: hot-delete %s: %v", entry.ModelName, err) + } + if err := c.hotAddLiteLLMModel(ctx, ns, entry); err != nil { + log.Printf("buyer-migrate: hot-add %s: %v; entry loads on next LiteLLM rollout", entry.ModelName, err) + } + } + log.Printf("buyer-migrate: rewrote %d legacy buyer api_base entries in %s/litellm-config", len(migrated), ns) +} + +// triggerBuyerReload sends POST /admin/reload to all running x402-buyer +// pods (a standalone Deployment since the litellm pod went stateless). +// Best-effort — the buyer reloads on its own 5-second timer anyway. func (c *Controller) triggerBuyerReload(ctx context.Context, ns string) { pods, err := c.kubeClient.CoreV1().Pods(ns).List(ctx, metav1.ListOptions{ - LabelSelector: "app=litellm", + LabelSelector: "app=x402-buyer", }) if err != nil || len(pods.Items) == 0 { return @@ -471,7 +573,7 @@ func (c *Controller) triggerBuyerReload(ctx context.Context, ns string) { func (c *Controller) triggerBuyerRemove(ctx context.Context, ns, name string) { pods, err := c.kubeClient.CoreV1().Pods(ns).List(ctx, metav1.ListOptions{ - LabelSelector: "app=litellm", + LabelSelector: "app=x402-buyer", }) if err != nil || len(pods.Items) == 0 { return @@ -495,12 +597,12 @@ func (c *Controller) triggerBuyerRemove(ctx context.Context, ns, name string) { // ── Sidecar status check ──────────────────────────────────────────────────── func (c *Controller) checkBuyerStatus(ctx context.Context, ns, name string) (remaining, spent int, err error) { - // List LiteLLM pods to get a pod IP. + // List x402-buyer pods to get a pod IP. pods, err := c.kubeClient.CoreV1().Pods(ns).List(ctx, metav1.ListOptions{ - LabelSelector: "app=litellm", + LabelSelector: "app=x402-buyer", }) if err != nil || len(pods.Items) == 0 { - return 0, 0, fmt.Errorf("no litellm pods in %s", ns) + return 0, 0, fmt.Errorf("no x402-buyer pods in %s", ns) } for _, pod := range pods.Items { diff --git a/internal/serviceoffercontroller/purchase_helpers_test.go b/internal/serviceoffercontroller/purchase_helpers_test.go index 21de7deb..c1732640 100644 --- a/internal/serviceoffercontroller/purchase_helpers_test.go +++ b/internal/serviceoffercontroller/purchase_helpers_test.go @@ -114,8 +114,8 @@ func TestAddLiteLLMModelEntryUpdatesConfigMapAndHotAdds(t *testing.T) { if entry.LiteLLMParams.Model != "openai/paid/qwen3.5:9b" { t.Fatalf("litellm_params.model = %q", entry.LiteLLMParams.Model) } - if entry.LiteLLMParams.APIBase != "http://127.0.0.1:8402/v1" { - t.Fatalf("litellm_params.api_base = %q", entry.LiteLLMParams.APIBase) + if entry.LiteLLMParams.APIBase != "http://x402-buyer.llm.svc.cluster.local:8402/v1" { + t.Fatalf("litellm_params.api_base = %q, want the x402-buyer Service (the buyer is no longer a litellm sidecar)", entry.LiteLLMParams.APIBase) } if got := fakeAPI.addCalls.Load(); got != 1 { @@ -348,18 +348,18 @@ func TestCheckBuyerStatusSkipsDeletingPods(t *testing.T) { kubeClient := fake.NewSimpleClientset( &corev1.Pod{ ObjectMeta: metav1.ObjectMeta{ - Name: "litellm-old", + Name: "x402-buyer-old", Namespace: "llm", - Labels: map[string]string{"app": "litellm"}, + Labels: map[string]string{"app": "x402-buyer"}, DeletionTimestamp: &now, }, Status: corev1.PodStatus{Phase: corev1.PodRunning, PodIP: "10.0.0.1"}, }, &corev1.Pod{ ObjectMeta: metav1.ObjectMeta{ - Name: "litellm-new", + Name: "x402-buyer-new", Namespace: "llm", - Labels: map[string]string{"app": "litellm"}, + Labels: map[string]string{"app": "x402-buyer"}, }, Status: corev1.PodStatus{Phase: corev1.PodRunning, PodIP: "10.0.0.2"}, }, diff --git a/internal/serviceoffercontroller/purchase_lifecycle_test.go b/internal/serviceoffercontroller/purchase_lifecycle_test.go index 65f96bd2..6ee31061 100644 --- a/internal/serviceoffercontroller/purchase_lifecycle_test.go +++ b/internal/serviceoffercontroller/purchase_lifecycle_test.go @@ -156,9 +156,9 @@ func newPurchaseLifecycleController(t *testing.T, purchases ...monetizeapi.Purch }, &corev1.Pod{ ObjectMeta: metav1.ObjectMeta{ - Name: "litellm-0", + Name: "x402-buyer-0", Namespace: "llm", - Labels: map[string]string{"app": "litellm"}, + Labels: map[string]string{"app": "x402-buyer"}, }, Status: corev1.PodStatus{ Phase: corev1.PodRunning, diff --git a/internal/serviceoffercontroller/purchase_migration_test.go b/internal/serviceoffercontroller/purchase_migration_test.go new file mode 100644 index 00000000..4df7eed5 --- /dev/null +++ b/internal/serviceoffercontroller/purchase_migration_test.go @@ -0,0 +1,201 @@ +package serviceoffercontroller + +import ( + "context" + "testing" + + "github.com/ObolNetwork/obol-stack/internal/model" + "gopkg.in/yaml.v3" + corev1 "k8s.io/api/core/v1" + metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + "k8s.io/client-go/kubernetes/fake" +) + +// These tests pin the upgrade path for the x402-buyer split (issue #321): +// clusters upgraded in place carry paid-route entries whose api_base still +// points at the removed litellm-pod sidecar (127.0.0.1:8402). Both the +// startup migration and the per-purchase reconcile must rewrite them to the +// x402-buyer Service and hot-sync the live router. + +func newMigrationController(t *testing.T, configYAML string) (*Controller, *litellmFake) { + t.Helper() + + cm := &corev1.ConfigMap{ + ObjectMeta: metav1.ObjectMeta{Name: "litellm-config", Namespace: "llm"}, + Data: map[string]string{"config.yaml": configYAML}, + } + secret := &corev1.Secret{ + ObjectMeta: metav1.ObjectMeta{Name: "litellm-secrets", Namespace: "llm"}, + Data: map[string][]byte{"LITELLM_MASTER_KEY": []byte("sk-obol-test")}, + } + fakeAPI := newLiteLLMFake() + t.Cleanup(fakeAPI.close) + + return &Controller{ + kubeClient: fake.NewSimpleClientset(cm, secret), + httpClient: fakeAPI.server.Client(), + litellmURLOverride: fakeAPI.server.URL, + }, fakeAPI +} + +func configMapModelList(t *testing.T, c *Controller) []model.ModelEntry { + t.Helper() + + cm, err := c.kubeClient.CoreV1().ConfigMaps("llm").Get(context.Background(), "litellm-config", metav1.GetOptions{}) + if err != nil { + t.Fatalf("get litellm-config: %v", err) + } + var cfg model.LiteLLMConfig + if err := yaml.Unmarshal([]byte(cm.Data["config.yaml"]), &cfg); err != nil { + t.Fatalf("parse config.yaml: %v", err) + } + return cfg.ModelList +} + +func TestBuyerAPIBase(t *testing.T) { + // The /v1 suffix is load-bearing: LiteLLM's OpenAI provider does not + // append it, and without it the buyer's mux returns 404 (CLAUDE.md + // pitfall 6). + if got, want := buyerAPIBase("llm"), "http://x402-buyer.llm.svc.cluster.local:8402/v1"; got != want { + t.Fatalf("buyerAPIBase(llm) = %q; want %q", got, want) + } +} + +func TestAddLiteLLMModelEntryMigratesLegacySidecarAPIBase(t *testing.T) { + const legacyConfig = `model_list: + - model_name: "paid/qwen36-deep" + litellm_params: + model: "openai/paid/qwen36-deep" + api_base: "http://127.0.0.1:8402/v1" + api_key: "unused" +` + c, fakeAPI := newMigrationController(t, legacyConfig) + fakeAPI.infoResp = []map[string]any{ + {"model_name": "paid/qwen36-deep", "model_info": map[string]any{"id": "id-1"}}, + } + + c.addLiteLLMModelEntry(context.Background(), "llm", "paid/qwen36-deep") + + entries := configMapModelList(t, c) + if len(entries) != 1 { + t.Fatalf("expected 1 entry after migration, got %d", len(entries)) + } + if got, want := entries[0].LiteLLMParams.APIBase, buyerAPIBase("llm"); got != want { + t.Errorf("api_base = %q; want migrated %q", got, want) + } + if got := fakeAPI.delCalls.Load(); got != 1 { + t.Errorf("delCalls = %d; want 1 (stale legacy deployment removed from live router)", got) + } + if got := fakeAPI.addCalls.Load(); got != 1 { + t.Errorf("addCalls = %d; want 1 (migrated entry hot-added)", got) + } +} + +func TestAddLiteLLMModelEntryLeavesNonLegacyEntryAlone(t *testing.T) { + const currentConfig = `model_list: + - model_name: "paid/qwen36-deep" + litellm_params: + model: "openai/paid/qwen36-deep" + api_base: "http://x402-buyer.llm.svc.cluster.local:8402/v1" + api_key: "unused" +` + c, fakeAPI := newMigrationController(t, currentConfig) + + c.addLiteLLMModelEntry(context.Background(), "llm", "paid/qwen36-deep") + + if got := fakeAPI.addCalls.Load() + fakeAPI.delCalls.Load(); got != 0 { + t.Errorf("expected no hot API calls for an up-to-date entry, got %d", got) + } + entries := configMapModelList(t, c) + if got, want := entries[0].LiteLLMParams.APIBase, buyerAPIBase("llm"); got != want { + t.Errorf("api_base = %q; want unchanged %q", got, want) + } +} + +func TestMigrateLegacyBuyerAPIBases(t *testing.T) { + // Wildcard and concrete legacy entries are both rewritten; entries + // pointing elsewhere (cloud providers, custom endpoints) are untouched. + const mixedConfig = `model_list: + - model_name: "paid/*" + litellm_params: + model: "openai/*" + api_base: "http://127.0.0.1:8402/v1" + api_key: "unused" + - model_name: "paid/qwen36-deep" + litellm_params: + model: "openai/paid/qwen36-deep" + api_base: "http://127.0.0.1:8402/v1" + api_key: "unused" + - model_name: "qwen36-deep" + litellm_params: + model: "openai/qwen36-deep" + api_base: "http://192.168.18.23:8000/v1" + api_key: "os.environ/CUSTOM_KEY" +` + c, fakeAPI := newMigrationController(t, mixedConfig) + fakeAPI.infoResp = []map[string]any{ + {"model_name": "paid/*", "model_info": map[string]any{"id": "id-wild"}}, + {"model_name": "paid/qwen36-deep", "model_info": map[string]any{"id": "id-1"}}, + } + + c.migrateLegacyBuyerAPIBases(context.Background(), "llm") + + entries := configMapModelList(t, c) + if len(entries) != 3 { + t.Fatalf("expected 3 entries, got %d", len(entries)) + } + byName := make(map[string]model.ModelEntry, len(entries)) + for _, e := range entries { + byName[e.ModelName] = e + } + for _, name := range []string{"paid/*", "paid/qwen36-deep"} { + if got, want := byName[name].LiteLLMParams.APIBase, buyerAPIBase("llm"); got != want { + t.Errorf("%s api_base = %q; want migrated %q", name, got, want) + } + } + if got, want := byName["qwen36-deep"].LiteLLMParams.APIBase, "http://192.168.18.23:8000/v1"; got != want { + t.Errorf("custom endpoint api_base = %q; want untouched %q", got, want) + } + + if got := fakeAPI.delCalls.Load(); got != 2 { + t.Errorf("delCalls = %d; want 2 (both legacy deployments removed live)", got) + } + if got := fakeAPI.addCalls.Load(); got != 2 { + t.Errorf("addCalls = %d; want 2 (both migrated entries hot-added)", got) + } +} + +func TestMigrateLegacyBuyerAPIBasesNoopWhenClean(t *testing.T) { + const cleanConfig = `model_list: + - model_name: "paid/*" + litellm_params: + model: "openai/*" + api_base: "http://x402-buyer.llm.svc.cluster.local:8402/v1" + api_key: "unused" +` + c, fakeAPI := newMigrationController(t, cleanConfig) + + c.migrateLegacyBuyerAPIBases(context.Background(), "llm") + + if got := fakeAPI.addCalls.Load() + fakeAPI.delCalls.Load(); got != 0 { + t.Errorf("expected no hot API calls on a clean config, got %d", got) + } +} + +func TestMigrateLegacyBuyerAPIBasesMissingConfigMap(t *testing.T) { + fakeAPI := newLiteLLMFake() + t.Cleanup(fakeAPI.close) + c := &Controller{ + kubeClient: fake.NewSimpleClientset(), + httpClient: fakeAPI.server.Client(), + litellmURLOverride: fakeAPI.server.URL, + } + + // Must not panic or call the API when litellm-config does not exist + // (fresh cluster where the controller starts before the chart applies). + c.migrateLegacyBuyerAPIBases(context.Background(), "llm") + + if got := fakeAPI.addCalls.Load() + fakeAPI.delCalls.Load(); got != 0 { + t.Errorf("expected no hot API calls without a ConfigMap, got %d", got) + } +} diff --git a/internal/stack/stack_test.go b/internal/stack/stack_test.go index 0854fda3..4244e16c 100644 --- a/internal/stack/stack_test.go +++ b/internal/stack/stack_test.go @@ -485,17 +485,19 @@ func TestLLMTemplate_IncludesPaidRouteAndBuyerSidecar(t *testing.T) { for _, want := range []string{ `model_name: "paid/*"`, `model: "openai/*"`, - `api_base: "http://127.0.0.1:8402/v1"`, + // Paid routes go through the standalone x402-buyer Service — the + // buyer is no longer a litellm-pod sidecar (issue #321: LiteLLM must + // be stateless so RollingUpdate maxUnavailable:0 gives zero-downtime + // rollouts). + `api_base: "http://x402-buyer.llm.svc.cluster.local:8402/v1"`, `name: x402-buyer`, `containerPort: 8402`, `name: buyer-http`, `name: x402-buyer-config`, `name: x402-buyer-auths`, - // litellm-config ONLY: the buyer ConfigMaps must stay out of the - // Reloader annotation — x402-buyer hot-reloads them via /admin/reload - // and the Recreate strategy would otherwise bounce the whole gateway - // on every buy/refill (CLAUDE.md pitfall 7). - `configmap.reloader.stakater.com/reload: "litellm-config"`, + // Key rotation is the one remaining case that needs a pod + // replacement (os.environ/ refs resolve at config load). + `secret.reloader.stakater.com/reload: "litellm-secrets"`, `emptyDir:`, } { if !strings.Contains(out, want) { @@ -503,8 +505,13 @@ func TestLLMTemplate_IncludesPaidRouteAndBuyerSidecar(t *testing.T) { } } - if strings.Contains(out, `configmap.reloader.stakater.com/reload: "litellm-config,`) { - t.Fatal("llm template reload annotation must list litellm-config only — buyer CM writes happen per purchase and would Recreate-bounce the gateway") + // Reloader must NOT watch litellm-config: every model_list change is + // hot-applied via /model/new + /model/delete (CLI and controller), and a + // ConfigMap-triggered rollout would reintroduce an inference gap on + // every model add/remove/prefer and first-time purchase (issue #321). + // The buyer ConfigMaps likewise hot-reload via /admin/reload. + if strings.Contains(out, "configmap.reloader.stakater.com/reload") { + t.Fatal("llm template must not carry a configmap Reloader annotation — model_list changes are hot-applied; a CM-triggered rollout gaps inference (issue #321)") } if strings.Contains(out, "custom_provider_map") {