feat(agent): improve readme, ci, resilience and skills#713
Open
OisinKyne wants to merge 9 commits into
Open
Conversation
Phase 1 of the zero-downtime work for #321: - obol model prefer no longer rolls the LiteLLM deployment. model_list order is an obol convention (Rank reads the ConfigMap); LiteLLM's router does not use it, so the restart was pure downtime. - RestartLiteLLM now fails loudly when the rollout does not converge within 90s instead of warning and reporting success (#321 item 3). - ReconcileRecorded's ConfigMap-only branch restarts explicitly now that Reloader no longer watches litellm-config. - New drift safety net: obol model status compares the ConfigMap model_list against the live router /v1/models (CheckRouterDrift / DiffRouterModels) and reports missing/extra entries, replacing the Reloader annotation as the guard against silently-failed hot calls. Claude-Session: https://claude.ai/code/session_01YLuXwN1A4tG3xmEMKvAzeD
…rollouts Phase 2 of #321. The buyer's RWO consumed-auth PVC forced the litellm Deployment onto replicas:1 + Recreate, turning every rollout (Reloader secret rotation, image bump, config reload) into a full inference gap — and the Reloader annotation on litellm-config triggered such a rollout on every model add/remove/prefer and first-time purchase, defeating the hot /model/new path shipped in #320. - x402-buyer becomes its own Deployment (1 replica, Recreate, keeps the PVC and single-writer auth semantics) + ClusterIP Service. Buyer CM changes still hot-reload via /admin/reload; only buyer image bumps briefly gap paid/* routes. - litellm is now stateless: RollingUpdate maxSurge:1 maxUnavailable:0 — a new pod must be Ready before the old one terminates. Reloader watches litellm-secrets only; litellm-config changes are hot-applied. - Paid-route api_base moves to http://x402-buyer.llm.svc.cluster.local:8402/v1. The controller migrates legacy 127.0.0.1:8402 entries at startup and on per-purchase reconcile (upgraded-in-place clusters), hot-syncing the live router. - Controller buyer probes (reload/remove/status) and buy.py target the x402-buyer pods; flow scripts reach the buyer via its Service. - PodMonitor keeps its historical name but targets the buyer pods. NetworkPolicy for :8402 is deliberately deferred: agent namespaces have no common label and buy.py legitimately reads /status from them; the Service changes addressing, not exposure (pod IP was already cluster-reachable). Claude-Session: https://claude.ai/code/session_01YLuXwN1A4tG3xmEMKvAzeD
CLAUDE.md and obol-stack-dev references: buyer is a standalone Deployment + Service (port-forward svc/x402-buyer), paid routes point at the buyer Service with the mandatory /v1 suffix, and Reloader watches litellm-secrets only — litellm-config changes are hot-applied with drift surfaced by obol model status (#321). Claude-Session: https://claude.ai/code/session_01YLuXwN1A4tG3xmEMKvAzeD
Replica-stance hardening for multi-node clusters, follow-up to the LiteLLM zero-downtime work (#321): - PodDisruptionBudgets (minAvailable: 1) for x402-verifier and eRPC — on multi-node clusters a kubectl drain would otherwise silently evict the only pod of the payment gate (every /services/* route 5xxs) or the RPC front door. Same stance as the existing litellm PDB; rollouts are unaffected (both surge gaplessly behind readiness probes). The eRPC PDB lives in base/templates/erpc.yaml because the ethereum/erpc chart (0.0.4) has no PDB template — its podDisruptionBudget value is unwired. cloudflared already ships a PDB gated on an active tunnel. - remote-signer pinned to strategy: Recreate post-helmfile-sync (hermes and openclaw runtimes). The obol/remote-signer chart (0.3.3) cannot express spec.strategy, so the deployment defaulted to RollingUpdate — a surge pod over the RWO keystore PVC wedges on multi-node volume attach and briefly double-runs the signer anywhere. Patching spec.strategy does not touch the pod template, so the pin never rolls the pod itself. Upstream follow-up: add strategy support to the chart. Reviewed and deliberately unchanged: serviceoffer-controller already has Lease-based leader election (cmd/serviceoffer-controller/main.go), so its RollingUpdate surge is safe — the new pod blocks on the Lease. Claude-Session: https://claude.ai/code/session_01YLuXwN1A4tG3xmEMKvAzeD
OisinKyne
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
v0.13.0-rc0 integration branch. This PR rolls the source PR stack and Renovate bumps into one release-candidate branch for validation against
integration/v0.13.0-rc0.The integration branch tip currently under validation is
a29c8262398afc63044d0fad1b57977286699ab4.Superseded source PRs
When this integration branch lands, these source PRs should be treated as superseded by the integrated RC branch rather than merged independently:
Conflict resolution
The source stacks overlapped in only two files during the initial rollup:
internal/serviceoffercontroller/controller.go— auto-merged cleanly.internal/embed/infrastructure/base/templates/llm.yaml— one conflict, resolved by keeping the LiteLLM PodDisruptionBudget. feat(app): resume installed apps on stack up + persisted value overrides + sell linkage #704 deleted it during cleanup; fix(infra): PDBs for payment/RPC gates + remote-signer Recreate strategy #708 established the PDB stance for payment/RPC gates, so the deletion was rejected.Validation
Local validation before release-smoke:
go build ./...go vet ./...bash -n flows/*.sh,git diff --check,go test ./internal/hermes -count=1RC smoke status before #715:
/api/services.jsonas a bare array.bedag/raw-2.0.2.tgzduring Hermes install.Targeted validation after #715 merge:
flow-13-dual-stack-obol.shwas rerun from integration tipa29c8262.alice-obol-inferencein/api/services.jsonwithHTTP 200 ... OBOL permit2, and the flow loggedPASSfor that catalog assertion.PurchaseRequestwas created after the retry. Treat that as the next blocker, separate from the fixed catalog-envelope bug.This PR is not smoke-green until flow-02 dependency preflights are clean and the full release smoke is rerun from the current integration tip.