Skip to content

fix(infra): PDBs for payment/RPC gates + remote-signer Recreate strategy#708

Open
bussyjd wants to merge 1 commit into
fix/litellm-zero-downtimefrom
fix/replica-stance-hardening
Open

fix(infra): PDBs for payment/RPC gates + remote-signer Recreate strategy#708
bussyjd wants to merge 1 commit into
fix/litellm-zero-downtimefrom
fix/replica-stance-hardening

Conversation

@bussyjd

@bussyjd bussyjd commented Jul 6, 2026

Copy link
Copy Markdown
Contributor

Replica-stance hardening: PDBs for payment/RPC gates + remote-signer Recreate

Follow-up to #707, from the replica-stance review of the remaining services. Targets multi-node clusters, where drains and RWO volume attach are real.

Stacked on #707 (fix/litellm-zero-downtime) — merge that first; this PR's own diff is the last commit.

Changes

PodDisruptionBudgets (minAvailable: 1)

  • x402-verifier (base/templates/x402.yaml) — the ForwardAuth gate for every paid route. On multi-node, a kubectl drain would silently evict the only pod and 5xx all /services/* traffic; the PDB makes that require explicit operator intent (delete the pod), same stance as the existing litellm PDB.
  • eRPC (base/templates/erpc.yaml) — the RPC front door for agents and the frontend. Lives in the base chart because the ethereum/erpc chart (0.0.4) has no PDB template; its podDisruptionBudget value in our values file is unwired. Selector pinned to the chart's live selectorLabels.
  • cloudflared — already ships a PDB (gated on active tunnel + >1 replica); now pinned by a test so it doesn't regress.

Rollouts are unaffected in all cases: at 1 replica the default RollingUpdate rounds to maxUnavailable 0 / maxSurge 1, and both verifier (/readyz gates on config + routes loaded) and eRPC surge gaplessly behind readiness probes.

remote-signer → strategy: Recreate

The obol/remote-signer chart (0.3.3) cannot express spec.strategy, so signer deployments defaulted to RollingUpdate — a singleton over a ReadWriteOnce keystore PVC. On multi-node, a surge pod can land on another node and wedge forever on the volume attach; on any cluster it briefly double-runs the signer over the same keystore. Same doctrine as hermes and x402-buyer: RWO-backed singletons run Recreate.

Implemented as a post-helmfile-sync kubectl patch (agentruntime.EnforceRemoteSignerRecreate) from both the hermes and openclaw sync paths, mirroring the existing hermes strategy-migration pattern — including the load-bearing explicit rollingUpdate: null that clears the k8s-defaulted block the API would otherwise reject the type flip over. Patching spec.strategy doesn't touch the pod template, so the pin never rolls the signer itself. Helm leaves it alone on later upgrades because the chart never renders the field.

Upstream follow-up: add strategy/updateStrategy support to the remote-signer chart in ObolNetwork/helm-charts, then delete this patch.

Reviewed and deliberately unchanged

  • serviceoffer-controller — flagged in the review as "RollingUpdate without leader election", but the controller already has Lease-based leader election (cmd/serviceoffer-controller/main.go:84, on by default), so its surge is safe: the new pod blocks on the Lease until the old leader releases it. No change; the x402.yaml comment already documents this.
  • Frontend readiness probe (cosmetic local-UI 502s during rollouts) — left for a UI-side change in obol-stack-front-end/chart.

Tests

  • internal/embed/embed_pdb_test.go — table-driven pins for both PDBs (namespace, minAvailable, exact selector — extra labels would silently match no pods) + cloudflared PDB template gating.
  • internal/agentruntime/signer_test.go — patch-args shape incl. the explicit rollingUpdate: null.

Live validation (multi-service k3d cluster from the #707 run)

  • obol stack up created both PDBs: erpc/erpc and x402/x402-verifier, MIN AVAILABLE 1, ALLOWED DISRUPTIONS 0.
  • The stack-up agent re-sync flipped remote-signer from RollingUpdate to {"type":"Recreate"} without rolling the pod (pod age preserved), confirming the patch is non-disruptive.
  • helm lint clean; full go test green on touched packages.

https://claude.ai/code/session_01YLuXwN1A4tG3xmEMKvAzeD

Replica-stance hardening for multi-node clusters, follow-up to the
LiteLLM zero-downtime work (#321):

- PodDisruptionBudgets (minAvailable: 1) for x402-verifier and eRPC —
  on multi-node clusters a kubectl drain would otherwise silently evict
  the only pod of the payment gate (every /services/* route 5xxs) or
  the RPC front door. Same stance as the existing litellm PDB; rollouts
  are unaffected (both surge gaplessly behind readiness probes). The
  eRPC PDB lives in base/templates/erpc.yaml because the ethereum/erpc
  chart (0.0.4) has no PDB template — its podDisruptionBudget value is
  unwired. cloudflared already ships a PDB gated on an active tunnel.

- remote-signer pinned to strategy: Recreate post-helmfile-sync (hermes
  and openclaw runtimes). The obol/remote-signer chart (0.3.3) cannot
  express spec.strategy, so the deployment defaulted to RollingUpdate —
  a surge pod over the RWO keystore PVC wedges on multi-node volume
  attach and briefly double-runs the signer anywhere. Patching
  spec.strategy does not touch the pod template, so the pin never rolls
  the pod itself. Upstream follow-up: add strategy support to the chart.

Reviewed and deliberately unchanged: serviceoffer-controller already
has Lease-based leader election (cmd/serviceoffer-controller/main.go),
so its RollingUpdate surge is safe — the new pod blocks on the Lease.

Claude-Session: https://claude.ai/code/session_01YLuXwN1A4tG3xmEMKvAzeD
@OisinKyne

OisinKyne commented Jul 6, 2026

Copy link
Copy Markdown
Contributor

Upstream follow-up: add strategy/updateStrategy support to the remote-signer chart in ObolNetwork/helm-charts, then delete this patch.

We control this chart, lets just fix this there rather than a post-install hook? I assume that would shorten install time

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants