feat: v0.13.0-rc0

.agents/skills/obol-stack-dev/SKILL.md

@@ -61,7 +61,7 @@ OBOL_TOKEN_BASE_SEPOLIA=0x0a09371a8b011d5110656ceBCc70603e53FD2c78
 
 **Payment assertion**: don't bypass the CLI/agent buy path with direct script exec. Prefer `obol buy inference`, `obol sell ...`, `obol agent ...`, and `obol kubectl` status checks. If the agent times out, diagnose Hermes/LiteLLM/model routing — don't relax the assertion. Required evidence: `PurchaseRequest Ready=True` + paid HTTP 200 + on-chain `Transfer` + exact balance deltas.
 
-**QA LLM**: full seller/buyer QA must route Alice and Bob through `OBOL_LLM_ENDPOINT` (OpenAI-compatible vLLM or llama.cpp on the QA host). Default `OBOL_LLM_MODEL=qwen36-deep` (27B-class). The smaller `qwen36-fast` (~4B) was the previous default but flakes on the long single-shot agent-buy prompt at flow-13/14 step 46 — see the retry-wrapper rationale in `flows/lib-dual-stack.sh::agent_buy_with_retry`. Sequence: `obol model setup custom` → `obol model prefer` → one `obol model sync`. Local Ollama and cloud-fallback are **not** acceptable green substitutes for full-flow QA.
+**QA LLM**: full seller/buyer QA must route Alice and Bob through the explicit `OBOL_LLM_ENDPOINT` and `OBOL_LLM_MODEL`. Treat the QA runner and the inference endpoint as separate inputs; never infer the endpoint from the runner hostname. Use an OpenAI-compatible model id returned by `/models`, and choose a long-prompt-capable model for flow-13/14 step 46. Sequence: `obol model setup custom` → `obol model prefer` → one `obol model sync`. Local Ollama and cloud-fallback are **not** acceptable green substitutes for full-flow QA.
 
 **Public vs private routes**: `/services/*`, `/.well-known/agent-registration.json`, `/skill.md`, and `/` (storefront) are public via the tunnel. **NEVER** remove `hostnames: ["obol.stack"]` from frontend or eRPC HTTPRoutes — exposing them publicly is a critical security flaw.
 
@@ -82,6 +82,9 @@ When the smoke gate goes red, check these first — each was a multi-hour debug:
 | Free-tier RPC 408 on balance reads | `drpc.org`/`sepolia.base.org` rate-limit aggressively under release-smoke load. | Set `BASE_SEPOLIA_RPC` to a paid drpc lb URL or `ALCHEMY_BASE_SEPOLIA_API_KEY`. `flows/release-smoke.sh` runs `warn_unpaid_base_sepolia_rpc` preflight. `flows/lib.sh::scrub_secrets` collapses paid-RPC URLs to TLD-only in logs. |
 | First request after fresh verifier deploy returns empty body | Traefik HTTPRoute is wired but verifier's serviceoffer-source watcher hasn't loaded the route yet. | `flows/flow-07-sell-verify.sh` + `flows/flow-08-buy.sh` — wrap 402-body fetch in 12×5s retry loop. |
 | facilitator arm64 image runs amd64 binary | Was an `ObolNetwork/x402-rs` prom-overlay arm64 manifest packaging bug. | **Fixed upstream**: `ObolNetwork/x402-rs#3` (merged 2026-05-13, `668b7bb`) dropped the redundant `--platform=$BUILDPLATFORM` pin from the prom-overlay builder stage. Registry image republished; arm64 manifest now ships an aarch64 ELF (digest `sha256:b209345c…`). The `X402_FACILITATOR_SKIP_PULL` knob has been removed from `flows/lib.sh`. |
+| flow-13 catalog assertion crashes with `'str' object has no attribute 'get'` | The smoke helper parsed `/api/services.json` as the old bare array and iterated object keys from the current catalog envelope. | `flows/lib-dual-stack.sh::assert_bob_service_catalog_contains` accepts both the current `{"services":[...]}` envelope and the legacy list form. |
+| Hermes install times out fetching `bedag/raw-2.0.2.tgz` | Hermes deploys generated Kubernetes resources through the third-party `bedag/raw` Helm chart. A chart repo/network miss blocks agent install even when the product logic is fine. | Classify as dependency/bootstrap drift. Longer-term debt reduction: replace the external raw chart with a small first-party/local raw-manifest chart or render/apply an embedded first-party Hermes chart. |
+| flow-02 stalls on `monitoring-kube-state-metrics` `ContainerCreating` | The kubelet is stuck pulling `registry.k8s.io/kube-state-metrics/kube-state-metrics:v2.18.0`. | Treat as registry/bootstrap drift. Preflight or pre-cache the exact image before full release-smoke; do not change flow assertions to hide it. |
 
 **Diagnosis pattern**: a 503 from the verifier or 404 from a paid route almost never means the verifier is bad — it usually means the deployed image isn't what you think it is, the chain id form mismatched, or the upstream wasn't reachable. Confirm the running image first (`obol kubectl get deploy -n x402 x402-verifier -o jsonpath='{.spec.template.spec.containers[*].image}'`) before diving into x402 logic.
 

.agents/skills/obol-stack-dev/references/llm-routing.md

@@ -24,10 +24,10 @@ Only routes published through Traefik are reachable at `http://obol.stack:8080/`
 |---|---|
 | Traefik ingress (frontend, eRPC, x402 routes) | `http://obol.stack:8080/...` |
 | LiteLLM | `obol kubectl port-forward svc/litellm 14000:4000 -n llm` then `http://127.0.0.1:14000` |
-| x402-buyer sidecar (no Service — pod only) | `obol kubectl port-forward -n llm <litellm-pod> 18402:8402` then `http://127.0.0.1:18402` |
+| x402-buyer (own Deployment + Service) | `obol kubectl port-forward -n llm svc/x402-buyer 18402:8402` then `http://127.0.0.1:18402` |
 | OpenClaw instance | `obol kubectl port-forward -n openclaw-<id> svc/openclaw 18789:18789` |
 
-`http://obol.stack:8080/v1/...` does **not** hit LiteLLM — Traefik has no `/v1` route and returns the frontend 404. The `x402-buyer` sidecar is **distroless** — no `wget`/`curl`/shell. Always port-forward, never `kubectl exec`.
+`http://obol.stack:8080/v1/...` does **not** hit LiteLLM — Traefik has no `/v1` route and returns the frontend 404. The `x402-buyer` binary is **distroless** — no `wget`/`curl`/shell. Always port-forward, never `kubectl exec`.
 
 ## Auto-Configuration During `stack up`
 
@@ -72,16 +72,16 @@ commands directly.
 LiteLLM has a static route added by the embedded config:
 
 ```
-paid/* → openai/* → http://127.0.0.1:8402/v1
+paid/* → openai/* → http://x402-buyer.llm.svc.cluster.local:8402/v1
 ```
 
-The `x402-buyer` sidecar in the litellm pod listens on port 8402.
+`x402-buyer` runs as its own Deployment + Service in the llm namespace, listening on port 8402 (split out of the litellm pod so LiteLLM rolls with maxUnavailable: 0 — issue #321).
 
 **Critical**: the trailing `/v1` is mandatory. LiteLLM's OpenAI provider does **not** append `/v1` to a bare `api_base`. Without it, LiteLLM calls `/chat/completions` on the buyer and the buyer mux returns Go's default `404 page not found`, surfaced as `OpenAIException - 404 page not found`.
 
 ### Buyer flow
 
-`buy.py` (in the agent pod, `${OBOL_SKILLS_DIR:-/data/.openclaw/skills}/buy-x402/scripts/buy.py`) creates a `PurchaseRequest` CR with pre-signed ERC-3009 (USDC) or Permit2 (OBOL) auths. The serviceoffer-controller reconciles the CR, writes per-upstream buyer config/auth files into the buyer ConfigMaps, and hot-adds the `paid/<model>` LiteLLM route. The sidecar spends one auth per paid request.
+`buy.py` (in the agent pod, `${OBOL_SKILLS_DIR:-/data/.openclaw/skills}/buy-x402/scripts/buy.py`) creates a `PurchaseRequest` CR with pre-signed ERC-3009 (USDC) or Permit2 (OBOL) auths. The serviceoffer-controller reconciles the CR, writes per-upstream buyer config/auth files into the buyer ConfigMaps, and hot-adds the `paid/<model>` LiteLLM route. The buyer spends one auth per paid request.
 
 ```
 probe <endpoint-url> [--model <id>]

.agents/skills/obol-stack-dev/references/release-smoke-debugging.md

@@ -122,6 +122,30 @@ The buyer sidecar's `/status` (and `PurchaseRequest.status`, and verifier logs)
 - **Operator debugging recipe** (when a buyer-reported "0 spent" disagrees with a suspected debit): see `docs/observability.md` § "Verify settlement against the chain, never the sidecar snapshot" — has the exact `eth_getLogs` curl to confirm.
 - **Rule of thumb**: chain is canonical, sidecar status is a derived snapshot. The CRD itself documents this (`PurchaseRequest.status` is the controller's last reconciled snapshot, not a live counter — `CLAUDE.md` "Quick full-cycle smoke test"). For real-time auth pool state, always query the sidecar `/status`; for real-money truth, always query the chain.
 
+### 12. flow-13 catalog assertion crashes on `/api/services.json`
+
+The public catalog is the versioned `ServiceCatalog` envelope (`{"services":[...]}`), not the old bare array. If a smoke helper iterates the decoded object directly, Python walks the string keys and crashes with `'str' object has no attribute 'get'`.
+
+- **Symptom**: flow-13 "discover Alice's OBOL service in `/api/services.json`" fails from Bob's agent pod even though the tunnel is reachable and the catalog route exists.
+- **Fix in repo**: `flows/lib-dual-stack.sh::assert_bob_service_catalog_contains` parses `payload["services"]` when the payload is an object and still accepts a legacy list for compatibility.
+- **If you see this again**: curl the catalog from the same pod and inspect the top-level JSON type before changing a flow script.
+
+### 13. Hermes depends on the third-party `bedag/raw` chart
+
+Hermes uses a generated Helmfile with `chart: bedag/raw`, currently pinned as `rawChartVersion = "2.0.2"` in `internal/hermes/hermes.go`. The chart renders the raw Kubernetes resources from `values-hermes.yaml` for the Hermes deployment/service/config, while `obol/remote-signer` is installed as a separate release.
+
+- **Symptom**: `obol hermes sync` or `obol agent sync` fails with a timeout fetching `https://github.com/bedag/helm-charts/releases/download/raw-2.0.2/raw-2.0.2.tgz`.
+- **Meaning**: this is a dependency/bootstrap failure, not proof that the smoke's payment or agent logic is broken.
+- **Debt reduction path**: replace `bedag/raw` with a first-party local chart embedded in obol-stack, or promote Hermes to a first-party chart so release smoke no longer depends on the Bedag chart repository during agent install.
+
+### 14. flow-02 waits on kube-state-metrics image pull
+
+The monitoring chart pulls `registry.k8s.io/kube-state-metrics/kube-state-metrics:v2.18.0`. If that pull stalls, `flow-02-stack-init-up` sits at "Core platform pods Running or Completed" until the poll timeout and reports the monitoring pod as `ContainerCreating`.
+
+- **Symptom**: all core pods except `monitoring-kube-state-metrics-*` are Running; `kubectl describe pod` shows only `Pulling image "registry.k8s.io/kube-state-metrics/kube-state-metrics:v2.18.0"`.
+- **Meaning**: this is registry/bootstrap drift unless the image is actually wrong for the platform.
+- **If you see this again**: preflight `docker pull registry.k8s.io/kube-state-metrics/kube-state-metrics:v2.18.0` or import a known-good cache into the k3d cluster before starting the full release smoke. Record the cache step as environment setup, not a product fix.
+
 ## Diagnostic Patterns
 
 - **Don't confuse 503 with "verifier broken"** — almost always one of #1, #2, #5, #6, or a missing CA bundle (`paid-flows.md`).

.agents/skills/obol-stack-dev/references/remote-qa.md

@@ -9,11 +9,16 @@ explicit release-smoke/full-flow gates or named flow regressions.
 - Never use the shared source checkout directly for QA.
 - Always create a separate worktree per run.
 - Assume parallel stack tests may already be running.
+- Treat the QA runner and LLM inference endpoint as separate inputs. Set
+  `OBOL_LLM_ENDPOINT` and `OBOL_LLM_MODEL` explicitly; do not infer either one
+  from the machine running the flow.
 - Never run broad host cleanup such as global Docker/k3d purges.
 - Delete only stacks whose stack IDs are recorded in the QA worktree.
 - Do not record hostnames, personal paths, or secrets in the skill.
 - Do not create custom shell scripts. Use `tmux` to run the exact CLI command
   sequence when a long run needs to continue unattended.
+- Never edit flow files inside a QA worktree to make a smoke pass. If the flow
+  is stale, fix it in the repo and rerun from that commit.
 
 ## Create Worktree
 
@@ -79,8 +84,9 @@ under `$QA/.tmp/` and avoid writing a wrapper script.
 
 ## Launch Release-Gate Flow
 
-Full seller/buyer QA needs an OpenAI-compatible endpoint on the QA machine.
-Set `OBOL_LLM_MODEL` to an id returned by `/models`.
+Full seller/buyer QA needs an OpenAI-compatible endpoint reachable from the QA
+runner and from the in-cluster LiteLLM pods. Set `OBOL_LLM_MODEL` to an id
+returned by `/models`.
 
 ```bash
 cd "$QA"
@@ -110,6 +116,24 @@ tmux has-session -t "qa-flow14-$ts" 2>/dev/null && echo RUNNING || echo NOT_RUNN
 tail -n 200 "$log"
 ```
 
+## Fast Failure Checks
+
+Before a full release smoke, spend two minutes ruling out environment drift:
+
+```bash
+curl -fsS "$OBOL_LLM_ENDPOINT/models" | head
+helm repo add bedag https://bedag.github.io/helm-charts/ >/dev/null 2>&1 || true
+helm pull bedag/raw --version 2.0.2 --destination "$QA/.tmp" >/dev/null
+docker pull registry.k8s.io/kube-state-metrics/kube-state-metrics:v2.18.0 >/dev/null
+k3d cluster list
+```
+
+If `helm pull bedag/raw --version 2.0.2` times out, Hermes agent install can
+fail before the flow reaches product logic. Record it as a dependency/bootstrap
+blocker and rerun after the repo or cache is healthy; do not mutate the flow.
+The kube-state-metrics pull catches the flow-02 monitoring dependency before a
+full smoke spends its first timeout waiting on a registry path.
+
 ## Cleanup
 
 ```bash

.agents/skills/obol-stack-dev/references/troubleshooting.md

@@ -61,7 +61,7 @@ obol kubectl create configmap ca-certificates -n x402 \
 
 ### `OpenAIException - 404 page not found` on `paid/<model>`
 
-`api_base` for the buyer route must end in `/v1`. LiteLLM's OpenAI provider does **not** append `/v1` to a bare `api_base`. The buyer route must be `http://127.0.0.1:8402/v1`, not `http://127.0.0.1:8402`.
+`api_base` for the buyer route must end in `/v1`. LiteLLM's OpenAI provider does **not** append `/v1` to a bare `api_base`. The buyer route must be `http://x402-buyer.llm.svc.cluster.local:8402/v1` (the standalone buyer Service), not the bare `:8402` address.
 
 ### `eRPC` `eth_call` cache lag
 

.github/workflows/lint-test.yaml

@@ -47,6 +47,41 @@ jobs:
       - name: Run chart-testing (install)
         run: ct install --target-branch ${{ github.event.repository.default_branch }}
 
+  go-test:
+    name: Go build, vet, and unit tests
+    runs-on: ubuntu-latest
+    steps:
+      - name: Checkout
+        uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
+
+      - name: Set up Go
+        uses: actions/setup-go@4a3601121dd01d1626a1e23e37211e3254c1c06c # v6.4.0
+        with:
+          go-version-file: 'go.mod'
+
+      - name: Build
+        run: go build ./...
+
+      - name: Vet
+        run: go vet ./...
+
+      - name: Unit tests
+        run: go test ./...
+
+  goreleaser-check:
+    name: goreleaser config valid
+    runs-on: ubuntu-latest
+    steps:
+      - name: Checkout
+        uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
+
+      - name: Check .goreleaser.yaml
+        uses: goreleaser/goreleaser-action@f06c13b6b1a9625abc9e6e439d9c05a8f2190e94 # v7.2.3
+        with:
+          distribution: goreleaser
+          version: '~> v2'
+          args: check
+
   generate-check:
     name: CRD generation up-to-date
     runs-on: ubuntu-latest