From d875552bda09832eecd07c257cbdb8dea1196acc Mon Sep 17 00:00:00 2001 From: bussyjd <145845+bussyjd@users.noreply.github.com> Date: Tue, 7 Jul 2026 19:21:37 +0400 Subject: [PATCH 1/2] test(smoke): harden rc dual-stack catalog checks --- .agents/skills/obol-stack-dev/SKILL.md | 4 ++- .../references/release-smoke-debugging.md | 16 ++++++++++++ .../obol-stack-dev/references/remote-qa.md | 25 +++++++++++++++++-- flows/flow-11-dual-stack.sh | 2 +- flows/flow-14-live-obol-base-sepolia.sh | 2 +- flows/lib-dual-stack.sh | 14 +++++++++-- 6 files changed, 56 insertions(+), 7 deletions(-) diff --git a/.agents/skills/obol-stack-dev/SKILL.md b/.agents/skills/obol-stack-dev/SKILL.md index dadfe8fb..284654bd 100644 --- a/.agents/skills/obol-stack-dev/SKILL.md +++ b/.agents/skills/obol-stack-dev/SKILL.md @@ -61,7 +61,7 @@ OBOL_TOKEN_BASE_SEPOLIA=0x0a09371a8b011d5110656ceBCc70603e53FD2c78 **Payment assertion**: don't bypass the CLI/agent buy path with direct script exec. Prefer `obol buy inference`, `obol sell ...`, `obol agent ...`, and `obol kubectl` status checks. If the agent times out, diagnose Hermes/LiteLLM/model routing — don't relax the assertion. Required evidence: `PurchaseRequest Ready=True` + paid HTTP 200 + on-chain `Transfer` + exact balance deltas. -**QA LLM**: full seller/buyer QA must route Alice and Bob through `OBOL_LLM_ENDPOINT` (OpenAI-compatible vLLM or llama.cpp on the QA host). Default `OBOL_LLM_MODEL=qwen36-deep` (27B-class). The smaller `qwen36-fast` (~4B) was the previous default but flakes on the long single-shot agent-buy prompt at flow-13/14 step 46 — see the retry-wrapper rationale in `flows/lib-dual-stack.sh::agent_buy_with_retry`. Sequence: `obol model setup custom` → `obol model prefer` → one `obol model sync`. Local Ollama and cloud-fallback are **not** acceptable green substitutes for full-flow QA. +**QA LLM**: full seller/buyer QA must route Alice and Bob through the explicit `OBOL_LLM_ENDPOINT` and `OBOL_LLM_MODEL`. Treat the QA runner and the inference endpoint as separate inputs; never infer the endpoint from the runner hostname. Use an OpenAI-compatible model id returned by `/models`, and choose a long-prompt-capable model for flow-13/14 step 46. Sequence: `obol model setup custom` → `obol model prefer` → one `obol model sync`. Local Ollama and cloud-fallback are **not** acceptable green substitutes for full-flow QA. **Public vs private routes**: `/services/*`, `/.well-known/agent-registration.json`, `/skill.md`, and `/` (storefront) are public via the tunnel. **NEVER** remove `hostnames: ["obol.stack"]` from frontend or eRPC HTTPRoutes — exposing them publicly is a critical security flaw. @@ -82,6 +82,8 @@ When the smoke gate goes red, check these first — each was a multi-hour debug: | Free-tier RPC 408 on balance reads | `drpc.org`/`sepolia.base.org` rate-limit aggressively under release-smoke load. | Set `BASE_SEPOLIA_RPC` to a paid drpc lb URL or `ALCHEMY_BASE_SEPOLIA_API_KEY`. `flows/release-smoke.sh` runs `warn_unpaid_base_sepolia_rpc` preflight. `flows/lib.sh::scrub_secrets` collapses paid-RPC URLs to TLD-only in logs. | | First request after fresh verifier deploy returns empty body | Traefik HTTPRoute is wired but verifier's serviceoffer-source watcher hasn't loaded the route yet. | `flows/flow-07-sell-verify.sh` + `flows/flow-08-buy.sh` — wrap 402-body fetch in 12×5s retry loop. | | facilitator arm64 image runs amd64 binary | Was an `ObolNetwork/x402-rs` prom-overlay arm64 manifest packaging bug. | **Fixed upstream**: `ObolNetwork/x402-rs#3` (merged 2026-05-13, `668b7bb`) dropped the redundant `--platform=$BUILDPLATFORM` pin from the prom-overlay builder stage. Registry image republished; arm64 manifest now ships an aarch64 ELF (digest `sha256:b209345c…`). The `X402_FACILITATOR_SKIP_PULL` knob has been removed from `flows/lib.sh`. | +| flow-13 catalog assertion crashes with `'str' object has no attribute 'get'` | The smoke helper parsed `/api/services.json` as the old bare array and iterated object keys from the current catalog envelope. | `flows/lib-dual-stack.sh::assert_bob_service_catalog_contains` accepts both the current `{"services":[...]}` envelope and the legacy list form. | +| Hermes install times out fetching `bedag/raw-2.0.2.tgz` | Hermes deploys generated Kubernetes resources through the third-party `bedag/raw` Helm chart. A chart repo/network miss blocks agent install even when the product logic is fine. | Classify as dependency/bootstrap drift. Longer-term debt reduction: replace the external raw chart with a small first-party/local raw-manifest chart or render/apply an embedded first-party Hermes chart. | **Diagnosis pattern**: a 503 from the verifier or 404 from a paid route almost never means the verifier is bad — it usually means the deployed image isn't what you think it is, the chain id form mismatched, or the upstream wasn't reachable. Confirm the running image first (`obol kubectl get deploy -n x402 x402-verifier -o jsonpath='{.spec.template.spec.containers[*].image}'`) before diving into x402 logic. diff --git a/.agents/skills/obol-stack-dev/references/release-smoke-debugging.md b/.agents/skills/obol-stack-dev/references/release-smoke-debugging.md index cc855569..7200351c 100644 --- a/.agents/skills/obol-stack-dev/references/release-smoke-debugging.md +++ b/.agents/skills/obol-stack-dev/references/release-smoke-debugging.md @@ -122,6 +122,22 @@ The buyer sidecar's `/status` (and `PurchaseRequest.status`, and verifier logs) - **Operator debugging recipe** (when a buyer-reported "0 spent" disagrees with a suspected debit): see `docs/observability.md` § "Verify settlement against the chain, never the sidecar snapshot" — has the exact `eth_getLogs` curl to confirm. - **Rule of thumb**: chain is canonical, sidecar status is a derived snapshot. The CRD itself documents this (`PurchaseRequest.status` is the controller's last reconciled snapshot, not a live counter — `CLAUDE.md` "Quick full-cycle smoke test"). For real-time auth pool state, always query the sidecar `/status`; for real-money truth, always query the chain. +### 12. flow-13 catalog assertion crashes on `/api/services.json` + +The public catalog is the versioned `ServiceCatalog` envelope (`{"services":[...]}`), not the old bare array. If a smoke helper iterates the decoded object directly, Python walks the string keys and crashes with `'str' object has no attribute 'get'`. + +- **Symptom**: flow-13 "discover Alice's OBOL service in `/api/services.json`" fails from Bob's agent pod even though the tunnel is reachable and the catalog route exists. +- **Fix in repo**: `flows/lib-dual-stack.sh::assert_bob_service_catalog_contains` parses `payload["services"]` when the payload is an object and still accepts a legacy list for compatibility. +- **If you see this again**: curl the catalog from the same pod and inspect the top-level JSON type before changing a flow script. + +### 13. Hermes depends on the third-party `bedag/raw` chart + +Hermes uses a generated Helmfile with `chart: bedag/raw`, currently pinned as `rawChartVersion = "2.0.2"` in `internal/hermes/hermes.go`. The chart renders the raw Kubernetes resources from `values-hermes.yaml` for the Hermes deployment/service/config, while `obol/remote-signer` is installed as a separate release. + +- **Symptom**: `obol hermes sync` or `obol agent sync` fails with a timeout fetching `https://github.com/bedag/helm-charts/releases/download/raw-2.0.2/raw-2.0.2.tgz`. +- **Meaning**: this is a dependency/bootstrap failure, not proof that the smoke's payment or agent logic is broken. +- **Debt reduction path**: replace `bedag/raw` with a first-party local chart embedded in obol-stack, or promote Hermes to a first-party chart so release smoke no longer depends on the Bedag chart repository during agent install. + ## Diagnostic Patterns - **Don't confuse 503 with "verifier broken"** — almost always one of #1, #2, #5, #6, or a missing CA bundle (`paid-flows.md`). diff --git a/.agents/skills/obol-stack-dev/references/remote-qa.md b/.agents/skills/obol-stack-dev/references/remote-qa.md index e52eb2c9..998ca4cd 100644 --- a/.agents/skills/obol-stack-dev/references/remote-qa.md +++ b/.agents/skills/obol-stack-dev/references/remote-qa.md @@ -9,11 +9,16 @@ explicit release-smoke/full-flow gates or named flow regressions. - Never use the shared source checkout directly for QA. - Always create a separate worktree per run. - Assume parallel stack tests may already be running. +- Treat the QA runner and LLM inference endpoint as separate inputs. Set + `OBOL_LLM_ENDPOINT` and `OBOL_LLM_MODEL` explicitly; do not infer either one + from the machine running the flow. - Never run broad host cleanup such as global Docker/k3d purges. - Delete only stacks whose stack IDs are recorded in the QA worktree. - Do not record hostnames, personal paths, or secrets in the skill. - Do not create custom shell scripts. Use `tmux` to run the exact CLI command sequence when a long run needs to continue unattended. +- Never edit flow files inside a QA worktree to make a smoke pass. If the flow + is stale, fix it in the repo and rerun from that commit. ## Create Worktree @@ -79,8 +84,9 @@ under `$QA/.tmp/` and avoid writing a wrapper script. ## Launch Release-Gate Flow -Full seller/buyer QA needs an OpenAI-compatible endpoint on the QA machine. -Set `OBOL_LLM_MODEL` to an id returned by `/models`. +Full seller/buyer QA needs an OpenAI-compatible endpoint reachable from the QA +runner and from the in-cluster LiteLLM pods. Set `OBOL_LLM_MODEL` to an id +returned by `/models`. ```bash cd "$QA" @@ -110,6 +116,21 @@ tmux has-session -t "qa-flow14-$ts" 2>/dev/null && echo RUNNING || echo NOT_RUNN tail -n 200 "$log" ``` +## Fast Failure Checks + +Before a full release smoke, spend two minutes ruling out environment drift: + +```bash +curl -fsS "$OBOL_LLM_ENDPOINT/models" | head +helm repo add bedag https://bedag.github.io/helm-charts/ >/dev/null 2>&1 || true +helm pull bedag/raw --version 2.0.2 --destination "$QA/.tmp" >/dev/null +k3d cluster list +``` + +If `helm pull bedag/raw --version 2.0.2` times out, Hermes agent install can +fail before the flow reaches product logic. Record it as a dependency/bootstrap +blocker and rerun after the repo or cache is healthy; do not mutate the flow. + ## Cleanup ```bash diff --git a/flows/flow-11-dual-stack.sh b/flows/flow-11-dual-stack.sh index fe092760..51b08a62 100755 --- a/flows/flow-11-dual-stack.sh +++ b/flows/flow-11-dual-stack.sh @@ -1133,7 +1133,7 @@ if [ -n "$REGISTRATION_TX" ] && receipt_status_ok "$REGISTRATION_TX"; then write_receipt registration "$REGISTRATION_TX" pass "Registration receipt archived: $REGISTRATION_TX" else - skip "Registration receipt unavailable for Agent ID $AGENT_ID (registration already reflected in ServiceOffer status)" + pass "Registration already reflected in ServiceOffer status; receipt lookup unavailable for Agent ID $AGENT_ID" fi if [ -n "$METADATA_TX" ] && receipt_status_ok "$METADATA_TX"; then write_receipt metadata "$METADATA_TX" diff --git a/flows/flow-14-live-obol-base-sepolia.sh b/flows/flow-14-live-obol-base-sepolia.sh index e357682a..df46ab3e 100755 --- a/flows/flow-14-live-obol-base-sepolia.sh +++ b/flows/flow-14-live-obol-base-sepolia.sh @@ -721,7 +721,7 @@ PY write_receipt registration "$REGISTRATION_TX" pass "Registration receipt archived: $REGISTRATION_TX" else - skip "Registration receipt unavailable for Agent ID $AGENT_ID (registration already reflected in ServiceOffer status)" + pass "Registration already reflected in ServiceOffer status; receipt lookup unavailable for Agent ID $AGENT_ID" fi if [ -n "$METADATA_TX" ] && receipt_status_ok "$METADATA_TX"; then write_receipt metadata "$METADATA_TX" diff --git a/flows/lib-dual-stack.sh b/flows/lib-dual-stack.sh index b8a72d76..8fcf7084 100644 --- a/flows/lib-dual-stack.sh +++ b/flows/lib-dual-stack.sh @@ -353,9 +353,19 @@ with urllib.request.urlopen( timeout=20, ) as resp: status = resp.status - services = json.loads(resp.read(200000)) + payload = json.loads(resp.read(200000)) -entry = next((svc for svc in services if svc.get("name") == service_name), None) +if isinstance(payload, dict): + services = payload.get("services", []) +elif isinstance(payload, list): + services = payload +else: + raise RuntimeError(f"unexpected catalog payload type {type(payload).__name__}") + +if not isinstance(services, list): + raise RuntimeError(f"unexpected catalog services type {type(services).__name__}") + +entry = next((svc for svc in services if isinstance(svc, dict) and svc.get("name") == service_name), None) if entry is None: raise RuntimeError(f"{service_name} not present") From 8160cfa2c22a3c7f1f3ee38bf9bff875347c6915 Mon Sep 17 00:00:00 2001 From: bussyjd <145845+bussyjd@users.noreply.github.com> Date: Tue, 7 Jul 2026 19:32:24 +0400 Subject: [PATCH 2/2] docs(qa): record smoke dependency preflights --- .agents/skills/obol-stack-dev/SKILL.md | 1 + .../obol-stack-dev/references/release-smoke-debugging.md | 8 ++++++++ .agents/skills/obol-stack-dev/references/remote-qa.md | 3 +++ 3 files changed, 12 insertions(+) diff --git a/.agents/skills/obol-stack-dev/SKILL.md b/.agents/skills/obol-stack-dev/SKILL.md index 284654bd..398372b2 100644 --- a/.agents/skills/obol-stack-dev/SKILL.md +++ b/.agents/skills/obol-stack-dev/SKILL.md @@ -84,6 +84,7 @@ When the smoke gate goes red, check these first — each was a multi-hour debug: | facilitator arm64 image runs amd64 binary | Was an `ObolNetwork/x402-rs` prom-overlay arm64 manifest packaging bug. | **Fixed upstream**: `ObolNetwork/x402-rs#3` (merged 2026-05-13, `668b7bb`) dropped the redundant `--platform=$BUILDPLATFORM` pin from the prom-overlay builder stage. Registry image republished; arm64 manifest now ships an aarch64 ELF (digest `sha256:b209345c…`). The `X402_FACILITATOR_SKIP_PULL` knob has been removed from `flows/lib.sh`. | | flow-13 catalog assertion crashes with `'str' object has no attribute 'get'` | The smoke helper parsed `/api/services.json` as the old bare array and iterated object keys from the current catalog envelope. | `flows/lib-dual-stack.sh::assert_bob_service_catalog_contains` accepts both the current `{"services":[...]}` envelope and the legacy list form. | | Hermes install times out fetching `bedag/raw-2.0.2.tgz` | Hermes deploys generated Kubernetes resources through the third-party `bedag/raw` Helm chart. A chart repo/network miss blocks agent install even when the product logic is fine. | Classify as dependency/bootstrap drift. Longer-term debt reduction: replace the external raw chart with a small first-party/local raw-manifest chart or render/apply an embedded first-party Hermes chart. | +| flow-02 stalls on `monitoring-kube-state-metrics` `ContainerCreating` | The kubelet is stuck pulling `registry.k8s.io/kube-state-metrics/kube-state-metrics:v2.18.0`. | Treat as registry/bootstrap drift. Preflight or pre-cache the exact image before full release-smoke; do not change flow assertions to hide it. | **Diagnosis pattern**: a 503 from the verifier or 404 from a paid route almost never means the verifier is bad — it usually means the deployed image isn't what you think it is, the chain id form mismatched, or the upstream wasn't reachable. Confirm the running image first (`obol kubectl get deploy -n x402 x402-verifier -o jsonpath='{.spec.template.spec.containers[*].image}'`) before diving into x402 logic. diff --git a/.agents/skills/obol-stack-dev/references/release-smoke-debugging.md b/.agents/skills/obol-stack-dev/references/release-smoke-debugging.md index 7200351c..c48b893a 100644 --- a/.agents/skills/obol-stack-dev/references/release-smoke-debugging.md +++ b/.agents/skills/obol-stack-dev/references/release-smoke-debugging.md @@ -138,6 +138,14 @@ Hermes uses a generated Helmfile with `chart: bedag/raw`, currently pinned as `r - **Meaning**: this is a dependency/bootstrap failure, not proof that the smoke's payment or agent logic is broken. - **Debt reduction path**: replace `bedag/raw` with a first-party local chart embedded in obol-stack, or promote Hermes to a first-party chart so release smoke no longer depends on the Bedag chart repository during agent install. +### 14. flow-02 waits on kube-state-metrics image pull + +The monitoring chart pulls `registry.k8s.io/kube-state-metrics/kube-state-metrics:v2.18.0`. If that pull stalls, `flow-02-stack-init-up` sits at "Core platform pods Running or Completed" until the poll timeout and reports the monitoring pod as `ContainerCreating`. + +- **Symptom**: all core pods except `monitoring-kube-state-metrics-*` are Running; `kubectl describe pod` shows only `Pulling image "registry.k8s.io/kube-state-metrics/kube-state-metrics:v2.18.0"`. +- **Meaning**: this is registry/bootstrap drift unless the image is actually wrong for the platform. +- **If you see this again**: preflight `docker pull registry.k8s.io/kube-state-metrics/kube-state-metrics:v2.18.0` or import a known-good cache into the k3d cluster before starting the full release smoke. Record the cache step as environment setup, not a product fix. + ## Diagnostic Patterns - **Don't confuse 503 with "verifier broken"** — almost always one of #1, #2, #5, #6, or a missing CA bundle (`paid-flows.md`). diff --git a/.agents/skills/obol-stack-dev/references/remote-qa.md b/.agents/skills/obol-stack-dev/references/remote-qa.md index 998ca4cd..1e740643 100644 --- a/.agents/skills/obol-stack-dev/references/remote-qa.md +++ b/.agents/skills/obol-stack-dev/references/remote-qa.md @@ -124,12 +124,15 @@ Before a full release smoke, spend two minutes ruling out environment drift: curl -fsS "$OBOL_LLM_ENDPOINT/models" | head helm repo add bedag https://bedag.github.io/helm-charts/ >/dev/null 2>&1 || true helm pull bedag/raw --version 2.0.2 --destination "$QA/.tmp" >/dev/null +docker pull registry.k8s.io/kube-state-metrics/kube-state-metrics:v2.18.0 >/dev/null k3d cluster list ``` If `helm pull bedag/raw --version 2.0.2` times out, Hermes agent install can fail before the flow reaches product logic. Record it as a dependency/bootstrap blocker and rerun after the repo or cache is healthy; do not mutate the flow. +The kube-state-metrics pull catches the flow-02 monitoring dependency before a +full smoke spends its first timeout waiting on a registry path. ## Cleanup