feat: v0.13.0-rc0#716
Open
OisinKyne wants to merge 27 commits into
Open
Conversation
Phase 1 of the zero-downtime work for #321: - obol model prefer no longer rolls the LiteLLM deployment. model_list order is an obol convention (Rank reads the ConfigMap); LiteLLM's router does not use it, so the restart was pure downtime. - RestartLiteLLM now fails loudly when the rollout does not converge within 90s instead of warning and reporting success (#321 item 3). - ReconcileRecorded's ConfigMap-only branch restarts explicitly now that Reloader no longer watches litellm-config. - New drift safety net: obol model status compares the ConfigMap model_list against the live router /v1/models (CheckRouterDrift / DiffRouterModels) and reports missing/extra entries, replacing the Reloader annotation as the guard against silently-failed hot calls. Claude-Session: https://claude.ai/code/session_01YLuXwN1A4tG3xmEMKvAzeD
…rollouts Phase 2 of #321. The buyer's RWO consumed-auth PVC forced the litellm Deployment onto replicas:1 + Recreate, turning every rollout (Reloader secret rotation, image bump, config reload) into a full inference gap — and the Reloader annotation on litellm-config triggered such a rollout on every model add/remove/prefer and first-time purchase, defeating the hot /model/new path shipped in #320. - x402-buyer becomes its own Deployment (1 replica, Recreate, keeps the PVC and single-writer auth semantics) + ClusterIP Service. Buyer CM changes still hot-reload via /admin/reload; only buyer image bumps briefly gap paid/* routes. - litellm is now stateless: RollingUpdate maxSurge:1 maxUnavailable:0 — a new pod must be Ready before the old one terminates. Reloader watches litellm-secrets only; litellm-config changes are hot-applied. - Paid-route api_base moves to http://x402-buyer.llm.svc.cluster.local:8402/v1. The controller migrates legacy 127.0.0.1:8402 entries at startup and on per-purchase reconcile (upgraded-in-place clusters), hot-syncing the live router. - Controller buyer probes (reload/remove/status) and buy.py target the x402-buyer pods; flow scripts reach the buyer via its Service. - PodMonitor keeps its historical name but targets the buyer pods. NetworkPolicy for :8402 is deliberately deferred: agent namespaces have no common label and buy.py legitimately reads /status from them; the Service changes addressing, not exposure (pod IP was already cluster-reachable). Claude-Session: https://claude.ai/code/session_01YLuXwN1A4tG3xmEMKvAzeD
CLAUDE.md and obol-stack-dev references: buyer is a standalone Deployment + Service (port-forward svc/x402-buyer), paid routes point at the buyer Service with the mandatory /v1 suffix, and Reloader watches litellm-secrets only — litellm-config changes are hot-applied with drift surfaced by obol model status (#321). Claude-Session: https://claude.ai/code/session_01YLuXwN1A4tG3xmEMKvAzeD
Replica-stance hardening for multi-node clusters, follow-up to the LiteLLM zero-downtime work (#321): - PodDisruptionBudgets (minAvailable: 1) for x402-verifier and eRPC — on multi-node clusters a kubectl drain would otherwise silently evict the only pod of the payment gate (every /services/* route 5xxs) or the RPC front door. Same stance as the existing litellm PDB; rollouts are unaffected (both surge gaplessly behind readiness probes). The eRPC PDB lives in base/templates/erpc.yaml because the ethereum/erpc chart (0.0.4) has no PDB template — its podDisruptionBudget value is unwired. cloudflared already ships a PDB gated on an active tunnel. - remote-signer pinned to strategy: Recreate post-helmfile-sync (hermes and openclaw runtimes). The obol/remote-signer chart (0.3.3) cannot express spec.strategy, so the deployment defaulted to RollingUpdate — a surge pod over the RWO keystore PVC wedges on multi-node volume attach and briefly double-runs the signer anywhere. Patching spec.strategy does not touch the pod template, so the pin never rolls the pod itself. Upstream follow-up: add strategy support to the chart. Reviewed and deliberately unchanged: serviceoffer-controller already has Lease-based leader election (cmd/serviceoffer-controller/main.go), so its RollingUpdate surge is safe — the new pod blocks on the Lease. Claude-Session: https://claude.ai/code/session_01YLuXwN1A4tG3xmEMKvAzeD
…v2026.7.1 The v2026.7.1 bump in this PR hardened the dashboard: a non-loopback (0.0.0.0) bind now REQUIRES an auth provider and the legacy --insecure flag no longer bypasses it (the hermes-0day fix). The master agent binds the dashboard on 0.0.0.0 for the k8s Service, so without this it crashloops (hermes-dashboard CrashLoopBackOff → deployment never Ready → flow-04 fail). Register the bundled basic-auth provider via env (user 'obol', password = the agent's existing API token); probe path /api/status stays auth-exempt. Child agents run 'gateway run' with no dashboard sidecar, so only the master agent (internal/hermes) needs it. Also update flow-04's dashboard check: the loopback-only inline __HERMES_SESSION_TOKEN__ is disabled on a gated bind, so assert the dashboard is up (public /api/status=200) and auth-gated (/api/sessions=401) instead. Validated on arm64 QA (spark): hermes pod 2/2 Running (was 1/2 CrashLoopBackOff); paid buy flow green.
…to integration/v0.13.0-rc0 # Conflicts: # internal/embed/infrastructure/base/templates/llm.yaml
…dates' into integration/v0.13.0-rc0
…dates' into integration/v0.13.0-rc0
…integration/v0.13.0-rc0
…nto integration/v0.13.0-rc0
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary