fix(infra): PDBs for payment/RPC gates + remote-signer Recreate strategy#708
Open
bussyjd wants to merge 1 commit into
Open
fix(infra): PDBs for payment/RPC gates + remote-signer Recreate strategy#708bussyjd wants to merge 1 commit into
bussyjd wants to merge 1 commit into
Conversation
Replica-stance hardening for multi-node clusters, follow-up to the LiteLLM zero-downtime work (#321): - PodDisruptionBudgets (minAvailable: 1) for x402-verifier and eRPC — on multi-node clusters a kubectl drain would otherwise silently evict the only pod of the payment gate (every /services/* route 5xxs) or the RPC front door. Same stance as the existing litellm PDB; rollouts are unaffected (both surge gaplessly behind readiness probes). The eRPC PDB lives in base/templates/erpc.yaml because the ethereum/erpc chart (0.0.4) has no PDB template — its podDisruptionBudget value is unwired. cloudflared already ships a PDB gated on an active tunnel. - remote-signer pinned to strategy: Recreate post-helmfile-sync (hermes and openclaw runtimes). The obol/remote-signer chart (0.3.3) cannot express spec.strategy, so the deployment defaulted to RollingUpdate — a surge pod over the RWO keystore PVC wedges on multi-node volume attach and briefly double-runs the signer anywhere. Patching spec.strategy does not touch the pod template, so the pin never rolls the pod itself. Upstream follow-up: add strategy support to the chart. Reviewed and deliberately unchanged: serviceoffer-controller already has Lease-based leader election (cmd/serviceoffer-controller/main.go), so its RollingUpdate surge is safe — the new pod blocks on the Lease. Claude-Session: https://claude.ai/code/session_01YLuXwN1A4tG3xmEMKvAzeD
Contributor
We control this chart, lets just fix this there rather than a post-install hook? I assume that would shorten install time |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Replica-stance hardening: PDBs for payment/RPC gates + remote-signer Recreate
Follow-up to #707, from the replica-stance review of the remaining services. Targets multi-node clusters, where drains and RWO volume attach are real.
Stacked on #707 (
fix/litellm-zero-downtime) — merge that first; this PR's own diff is the last commit.Changes
PodDisruptionBudgets (minAvailable: 1)
base/templates/x402.yaml) — the ForwardAuth gate for every paid route. On multi-node, akubectl drainwould silently evict the only pod and 5xx all/services/*traffic; the PDB makes that require explicit operator intent (delete the pod), same stance as the existing litellm PDB.base/templates/erpc.yaml) — the RPC front door for agents and the frontend. Lives in the base chart because theethereum/erpcchart (0.0.4) has no PDB template; itspodDisruptionBudgetvalue in our values file is unwired. Selector pinned to the chart's liveselectorLabels.Rollouts are unaffected in all cases: at 1 replica the default RollingUpdate rounds to maxUnavailable 0 / maxSurge 1, and both verifier (
/readyzgates on config + routes loaded) and eRPC surge gaplessly behind readiness probes.remote-signer → strategy: Recreate
The
obol/remote-signerchart (0.3.3) cannot expressspec.strategy, so signer deployments defaulted to RollingUpdate — a singleton over a ReadWriteOnce keystore PVC. On multi-node, a surge pod can land on another node and wedge forever on the volume attach; on any cluster it briefly double-runs the signer over the same keystore. Same doctrine as hermes and x402-buyer: RWO-backed singletons run Recreate.Implemented as a post-helmfile-sync
kubectl patch(agentruntime.EnforceRemoteSignerRecreate) from both the hermes and openclaw sync paths, mirroring the existing hermes strategy-migration pattern — including the load-bearing explicitrollingUpdate: nullthat clears the k8s-defaulted block the API would otherwise reject the type flip over. Patchingspec.strategydoesn't touch the pod template, so the pin never rolls the signer itself. Helm leaves it alone on later upgrades because the chart never renders the field.Upstream follow-up: add
strategy/updateStrategysupport to theremote-signerchart inObolNetwork/helm-charts, then delete this patch.Reviewed and deliberately unchanged
cmd/serviceoffer-controller/main.go:84, on by default), so its surge is safe: the new pod blocks on the Lease until the old leader releases it. No change; the x402.yaml comment already documents this.obol-stack-front-end/chart.Tests
internal/embed/embed_pdb_test.go— table-driven pins for both PDBs (namespace, minAvailable, exact selector — extra labels would silently match no pods) + cloudflared PDB template gating.internal/agentruntime/signer_test.go— patch-args shape incl. the explicitrollingUpdate: null.Live validation (multi-service k3d cluster from the #707 run)
obol stack upcreated both PDBs:erpc/erpcandx402/x402-verifier,MIN AVAILABLE 1,ALLOWED DISRUPTIONS 0.remote-signerfrom RollingUpdate to{"type":"Recreate"}without rolling the pod (pod age preserved), confirming the patch is non-disruptive.helm lintclean; fullgo testgreen on touched packages.https://claude.ai/code/session_01YLuXwN1A4tG3xmEMKvAzeD