diff --git a/changelog.d/pr4-ramp-rollback-runbooks-comparison-script.added.md b/changelog.d/pr4-ramp-rollback-runbooks-comparison-script.added.md new file mode 100644 index 000000000..3c2263796 --- /dev/null +++ b/changelog.d/pr4-ramp-rollback-runbooks-comparison-script.added.md @@ -0,0 +1 @@ +Add the ramp and rollback runbooks for the Cloud Run host cutover (logs-first gate checklist, three rollback classes) and scripts/compare_migration_baseline.py, which gates a candidate baseline against a reference baseline per probe (error rate ≤ +0.1pp, p95 ≤ ×1.20) for objective ramp decisions. diff --git a/changelog.d/pr4-stage5-validation-runbook.added.md b/changelog.d/pr4-stage5-validation-runbook.added.md new file mode 100644 index 000000000..5d744f2bc --- /dev/null +++ b/changelog.d/pr4-stage5-validation-runbook.added.md @@ -0,0 +1 @@ +Add the Stage 5 validation runbook for the Cloud Run host cutover: full-path checks via the api-lb hostname, the long-request proof, and the 50/50 weighted-routing test. Documentation only. diff --git a/docs/migration/history/pr4-stage5-lb-validation-runbook.md b/docs/migration/history/pr4-stage5-lb-validation-runbook.md new file mode 100644 index 000000000..deb44f907 --- /dev/null +++ b/docs/migration/history/pr4-stage5-lb-validation-runbook.md @@ -0,0 +1,218 @@ +# PR 4 Stage 5 Runbook: Full-Path Validation via api-lb.policyengine.org + +> Per-stage record: every command for the Stage 5 validation, committed before +> execution. Current operational behavior: [`../cloud-run-operations.md`](../cloud-run-operations.md) +> (the "URL map weight changes" section there is the procedure steps 4–5 rely on — +> merged with the Stage 4 PR). Source of truth for gates: the Stage 5 section of +> `api-v1-pr4-cloud-run-host-cutover-execution-plan.md` in the planning folder. + +**The second-hostname trick:** `api-lb.policyengine.org` resolves to the load balancer +while `api.policyengine.org` keeps pointing at App Engine. The LB gets a real, +TLS-valid, publicly-resolvable name whose only clients are us — so every experiment in +this stage, including the 50/50 split, is invisible to actual users. The Stage 4 cert +already covers `api-lb`; no certificate work here. + +Inputs from Stage 4 execution: `IP4 = 34.117.200.13`, `IP6 = 2600:1901:0:347b::`, +URL map `lb-api`, backends `bs-app-engine` / `bs-cloud-run`, snapshot procedure in the +ops doc. Step 4 is the **first time Cloud Run ever serves through the LB** — its warm +instance (service-level min 1) and maxScale 4 are already in place from Stage 3. + +## 0. Shell variables + +```bash +export PROJECT=policyengine-api +export LB_HOST=api-lb.policyengine.org +export GAE_HOST=api.policyengine.org +export SNAP_DIR=docs/migration/urlmap +``` + +## 1. DNS: `api-lb` records at Squarespace (manual, one-time) + +Add two records for `policyengine.org`: + +| Host | Type | Data | +|---|---|---| +| `api-lb` | A | `34.117.200.13` | +| `api-lb` | AAAA | `2600:1901:0:347b::` | + +Additive only — nothing about `api` changes. Verify propagation: + +```bash +dig +short A "$LB_HOST" @8.8.8.8 # expect 34.117.200.13 +dig +short AAAA "$LB_HOST" @8.8.8.8 # expect 2600:1901:0:347b:: +curl -s -o /dev/null -w '%{http_code}\n' "https://$LB_HOST/liveness-check" # expect 200 +``` + +## 2. Full-path validation at 100/0 (LB overhead quantification) + +Both captures in the same hour, backends identical (GAE serves both paths), so any +delta is pure LB overhead. Warm both paths with a few throwaway requests first. + +```bash +python scripts/capture_migration_baseline.py --base-url "https://$LB_HOST" \ + --repetitions 10 > /tmp/s5-lb-baseline.json +python scripts/capture_migration_baseline.py --base-url "https://$GAE_HOST" \ + --repetitions 10 > /tmp/s5-gae-baseline.json +# GATE: per probe, LB p50 <= direct-GAE p50 + 50ms; error rate 0 on both. + +API_BASE_URL="https://$LB_HOST" python -m pytest \ + tests/integration/test_live_calculate.py \ + tests/integration/test_live_economy.py \ + tests/integration/test_live_budget_window_cache.py -v +# GATE: all green through the LB at 100/0. +``` + +## 3. Long-request proof (validates the fixed 60-min serverless-NEG timeout) + +Stage 4 execution established (docs + API error) that serverless-NEG backends ignore +`timeoutSec` and get a fixed 60-minute LB timeout. This step proves it on our LB: a +cache-busted engine calculate runs ~40s+ — well past the displayed-but-ignored 30s. + +```bash +python3 - <<'EOF' > /tmp/s5-long-calc.json +import json +p = json.load(open("tests/data/calculate_us_1_data.json")) +p["_loadtest"] = "s5-long-request-proof" # cache-bust nonce (full-body hash key) +json.dump(p, open("/tmp/s5-long-calc.json", "w")) +EOF +curl -s -o /dev/null -w 'status=%{http_code} time=%{time_total}s\n' --max-time 290 \ + -X POST "https://api-lb.policyengine.org/us/calculate" \ + -H 'Content-Type: application/json' -d @/tmp/s5-long-calc.json +# GATE: status=200 with time well over 30s (typically ~40-60s). A 502/504 near +# 30s would mean the LB is cutting long requests — stop and investigate. +``` + +(The economy flow in step 2's suite doubles as a second long-flow proof: its job +computation runs minutes end-to-end, though via polling rather than one long request.) + +## 4. The 50/50 weighted-routing proof (first Cloud Run traffic through the LB) + +Weight change strictly via the ops-doc procedure — export, **diff against the last +committed snapshot**, edit only the two weights, import, verify readback: + +```bash +gcloud compute url-maps export lb-api --project "$PROJECT" --destination /tmp/lb-api-50.yaml --quiet +diff "$SNAP_DIR/$(ls $SNAP_DIR | tail -1)" /tmp/lb-api-50.yaml # expect fingerprint-only noise +# edit: weight: 100 -> 50 (bs-app-engine), weight: 0 -> 50 (bs-cloud-run) +gcloud compute url-maps import lb-api --project "$PROJECT" --source /tmp/lb-api-50.yaml --quiet +gcloud compute url-maps describe lb-api --project "$PROJECT" --format 'yaml(defaultRouteAction)' +cp /tmp/lb-api-50.yaml "$SNAP_DIR/$(date -u +%Y%m%dT%H%M%SZ)-lb-api-5050.yaml" +``` + +Routing distribution — 200 header-sampled requests: + +```bash +for i in $(seq 1 200); do + curl -sI "https://$LB_HOST/liveness-check" | grep -i x-policyengine-backend +done | sort | uniq -c +# GATE: cloud_run share in [35%, 65%] of 200; zero requests without the header. +``` + +Then the live suites again at 50/50 — this is the **cross-backend job polling** proof: +economy flows POST a job to one backend and poll it from whichever backend each poll +lands on, working only because both share Cloud SQL state: + +```bash +API_BASE_URL="https://$LB_HOST" python -m pytest \ + tests/integration/test_live_calculate.py \ + tests/integration/test_live_economy.py \ + tests/integration/test_live_budget_window_cache.py -v +# GATE: green at 50/50; zero 5xx in LB logs for the window: +gcloud logging read 'resource.type="http_load_balancer" + resource.labels.backend_service_name=("bs-app-engine" OR "bs-cloud-run") + httpRequest.status>=500 timestamp>=""' --project "$PROJECT" --limit 10 +``` + +While here, spot-check `X-Forwarded-For` handling (requests now arrive via a proxy on +both backends): sample an app log line on each backend and confirm the client IP is the +real client, not the LB's. + +## 5. Revert to 100/0 and commit both snapshots + +```bash +# same procedure: export current (the 50/50), edit weights back to 100/0, import +gcloud compute url-maps import lb-api --project "$PROJECT" --source /tmp/lb-api-100.yaml --quiet +for i in $(seq 1 200); do + curl -sI "https://$LB_HOST/liveness-check" | grep -i x-policyengine-backend +done | sort | uniq -c +# GATE: zero cloud_run of 200 — weights verified restored. +cp /tmp/lb-api-100.yaml "$SNAP_DIR/$(date -u +%Y%m%dT%H%M%SZ)-lb-api-restored.yaml" +git add "$SNAP_DIR" && git commit -m "Snapshot LB URL map (Stage 5: 50/50 test + restore to 100/0)" +``` + +## Exit gates (from the plan) + +- Suites green through the LB at 100/0 **and** 50/50; cross-backend polling works. +- LB overhead quantified: per-probe p50 ≤ direct-GAE p50 + 50ms; error rate 0. +- Long-request proof: >30s calculate completes (200), LB never the binding constraint. +- 50/50 header counts within [35%, 65%] of 200 requests; zero 5xx during the split. +- Weights verified restored to 100/0 (200 curls, zero `cloud_run`). +- Both URL-map snapshots committed. + +## Notes and blast radius + +- Users are untouched throughout: only `api-lb` resolves to the LB, and its only + clients are these validation commands. +- Step 4 sends real (validation) traffic to the production Cloud Run service for the + first time via the LB — expected fine: warm instance, maxScale 4, the same suites + pass against its direct URL every deploy. +- If anything fails mid-50/50, the immediate remedy is step 5's revert (weights back + to 100/0); the `api-lb` DNS records can also simply be removed. +- Rough duration: 1–2 hours, dominated by two integration-suite runs. + +## Execution record + +### 2026-07-08 — first execution (all steps) + +- Step 1: `api-lb` A/AAAA added at Squarespace; propagation + TLS verified. +- Step 2 (overhead gate): **PASS** — the LB path was *faster* than direct GAE on every + probe (p50 deltas −290 to −644ms; modern GFE front end beats `ghs.googlehosted.com`). + Suites green at 100/0 (one budget-window flake, passed on isolated retry — known + backend queue contention). +- Step 3 (long-request proof): **PASS** — cache-busted calculate returned 200 well past + 30s; calculate runtime is payload-insensitive (~28–31s on GAE), so the economy suite's + minutes-long polling flow doubles as the long-flow proof. +- Step 4 (50/50): distribution in bounds; suites 7/7 — **cross-backend job polling + proven** (POST lands on one backend, polls served by both, shared Cloud SQL). + **Caught one real bug:** the first Cloud Run scale-out (1→3 instances) served three + 503s — a two-worker cold-boot race on local sqlite init (`UNIQUE constraint failed: + policy.id`; silent variant: a worker boots seeing 0/2 seed rows). Issue api#3746, + fixed by an fcntl file lock around the exists-check + initialize in api#3747 + (verified with a 6-process fork-model boot harness: unfixed fails 3/3 rounds, fixed + clean), merged and deployed the same day. +- Step 5: reverted to 100/0, verified 100/100 header samples `app_engine`. Observed: + **URL-map weight changes take ~5–10 min to propagate across GFEs** — `describe` + shows the new weights immediately, but sampling reflects them only after the settle + period. Every future ramp step must wait out propagation before its sampling gate. + +### 2026-07-09 — zero-5xx re-run against the fixed revision (12:53–13:30Z) + +Purpose: yesterday's zero-5xx gate failed on the sqlite race; re-run scale-out boots on +the fixed revision (`policyengine-api-00058-sib`, the api#3747 deploy). + +- 50/50 via the ops-doc procedure (12:53:44Z); propagation confirmed at 12:57Z. +- Load: `scripts/measure_cloud_run_runtime.py` through the LB — 16 closed-loop clients, + 10 min, `--cache-bust always` (every calculate does real engine work, ~30–45s). This + held ~8 concurrent engine calls on the Cloud Run side (~4× one instance's + uncached-calculate capacity) and forced a **1→4 instance scale-out**. +- **Sqlite race confirmed fixed:** zero 503s, zero tracebacks/`UNIQUE constraint` + errors, zero worker crashes across all fresh boots. Client-side: 82/82 measured + calculates 200; one 429 (Cloud Run queue backpressure, expected at saturation); zero + 5xx on `bs-app-engine` for the whole window. +- **Finding — three 504s on `bs-cloud-run`** (12:59:11Z, LB latencies ~306s, Cloud Run + latencies ~299.97s, `response_sent_by_backend`): all three were calculates routed to + the *first* scale-out instance at its birth — the instance's request logs for them + precede its own "Starting new instance" line. They queued through the ~170s app + import (the accepted early-bind routing-before-ready tradeoff), then ran + GIL-contended and hit **Cloud Run's fixed 300s request timeout**. Later boots + absorbed their queued traffic within the timeout. The load generator's stats stayed + clean because the trio fired at the warmup/measurement boundary (excluded bucket) and + was abandoned client-side at the 290s client timeout. +- **Gate disposition (user-approved):** recorded as a known cold-scale-out saturation + artifact, not a defect — the zero-5xx gate is interpreted as zero *boot-failure* 5xx + and Stage 5 is **CLOSED**. Carry-forward for Stage 6 ramp monitoring: isolated 504 + clusters coinciding with instance starts are cold-routing, distinct from steady-state + 5xx. +- Reverted to 100/0 (13:17Z), verified 200/200 header samples `app_engine` by 13:30Z. + Both URL-map snapshots committed (`*-lb-api-5050-stage5-rerun.yaml`, + `*-lb-api-restored-100-0-rerun.yaml`). diff --git a/docs/migration/pr4-app-engine-rollback-runbook.md b/docs/migration/pr4-app-engine-rollback-runbook.md new file mode 100644 index 000000000..a30a3c67c --- /dev/null +++ b/docs/migration/pr4-app-engine-rollback-runbook.md @@ -0,0 +1,96 @@ +# PR 4 Rollback Runbook: returning traffic to App Engine + +> Live operational document during the ramp campaign. Three rollback classes, fastest +> first. **Class (a) is the failover mechanism for this architecture**: serverless NEGs +> have no LB health checks, so nothing fails over automatically — a human runs (a). +> Weight-change mechanics: [`cloud-run-operations.md`](cloud-run-operations.md) +> ("URL map weight changes"). + +## Which class for which symptom + +| Symptom | Class | +|---|---| +| 5xx / latency regression on the Cloud Run path; failed ramp gate; Cloud Run capacity or scaling trouble | **(a)** weights | +| Bad application release (breaks both backends — same code deploys to both) | **(a)** first if Cloud-Run-skewed, then **(b)** GAE version rollback | +| LB-layer failure: cert expiry/misissue, forwarding-rule or URL-map misconfiguration that can't be fixed forward in minutes | **(c)** DNS — last resort, time-boxed | + +## Class (a) — primary: URL-map weights → App Engine 100/0 (~minutes) + +Export → edit → import per the ops-doc procedure (abbreviated here for speed; diff +against the last snapshot when time permits, skip when firefighting): + +```bash +gcloud compute url-maps export lb-api --project policyengine-api \ + --destination /tmp/rollback.yaml --quiet +# edit: bs-app-engine weight → 100, bs-cloud-run weight → 0 +gcloud compute url-maps import lb-api --project policyengine-api \ + --source /tmp/rollback.yaml --quiet +gcloud compute url-maps describe lb-api --project policyengine-api \ + --format 'yaml(defaultRouteAction)' +``` + +- Control plane applies immediately; **GFEs follow over ~5–10 min** (observed Stages + 4–5). Verify with header sampling, not `describe`: + +```bash +for i in $(seq 1 200); do + curl -sI --max-time 10 https://api.policyengine.org/liveness-check | tr -d '\r' \ + | awk -F': ' 'tolower($1)=="x-policyengine-backend" {print $2}' +done | sort | uniq -c # expect 200 app_engine after settle +``` + +- Afterward: commit a timestamped snapshot to `docs/migration/urlmap/`, sweep the LB + logs for the incident window (filter `bs-*` backend names, not `aef-*`), and record + the incident in `docs/migration/history/`. +- App Engine has served 100% until this campaign and is never scaled down during it + (Stage 11 decommission happens only after the post-100% soak), so (a) needs no + capacity preparation. + +## Class (b) — App Engine version rollback (existing mechanism, unchanged) + +For a bad application release rather than a path problem. Works regardless of LB +weights, since it acts on the GAE service behind `bs-app-engine`: + +```bash +gcloud app versions list --project policyengine-api --service default \ + --sort-by '~version.createTime' | head # identify last-good version +gcloud app services set-traffic default \ + --splits =1 --project policyengine-api --quiet +``` + +Pair with class (a) so the rolled-back GAE version actually receives the traffic. Note +the Cloud Run side keeps serving the bad release until a fix-forward deploy — another +reason (a) accompanies (b). + +## Class (c) — catastrophic: DNS back to pre-cutover records (time-boxed!) + +Only for LB-layer failures that can't be fixed forward. Restore the pre-cutover +records at Squarespace (shape: `api` CNAME `ghs.googlehosted.com.`). Verbatim values +are deliberately NOT embedded here — get them from, in order: + +1. **Authoritative, always current:** `gcloud app domain-mappings describe + api.policyengine.org --project policyengine-api` (or App Engine console → Settings + → Custom domains) — shows exactly the records App Engine expects. Valid until the + domain mapping is deleted at Stage 11 decommission. +2. The pre-cutover capture (records + TTLs, taken at Stage 7) in the private migration + planning folder. + +Two clocks limit this option: + +1. **TTL**: propagation is bounded by the TTL recorded at Stage 7 (Squarespace floor; + expect ≥1h). Clients resolve back gradually, not at once. +2. **Certificate decay**: once DNS moves to the LB at Stage 8, App Engine's managed + certificate for `api.policyengine.org` stops renewing. Record its expiry at cutover + (typically ≤90 days out). **After that expiry, class (c) serves invalid TLS and is + dead** — the only path is fixing the LB forward and using class (a). + +After any (c) execution: the LB keeps running (no teardown under pressure); re-cutover +follows the Stage 8 procedure once the failure is understood. + +## After any rollback + +1. Snapshot + commit the URL map state (even for (b)/(c) — record what was serving). +2. Preserve evidence: LB log sweep and Cloud Run service logs for the incident window. +3. Write the incident record in `docs/migration/history/` before resuming the ramp; + resume from step 1 of the ramp ladder gate sequence at the weight you rolled back + from, not where you left off, if the cause was load-correlated. diff --git a/docs/migration/pr4-ramp-runbook.md b/docs/migration/pr4-ramp-runbook.md new file mode 100644 index 000000000..35e55edeb --- /dev/null +++ b/docs/migration/pr4-ramp-runbook.md @@ -0,0 +1,151 @@ +# PR 4 Ramp Runbook: App Engine → Cloud Run traffic ramp + +> Repeated-use operational document — one pass per ramp step. Executed after Stage 8 +> (DNS on the LB, weights 100/0). Weight-change mechanics live in +> [`cloud-run-operations.md`](cloud-run-operations.md) ("URL map weight changes"); +> rollback in [`pr4-app-engine-rollback-runbook.md`](pr4-app-engine-rollback-runbook.md). +> Gates are **logs-first**: both backend services log at `sampleRate: 1.0` (verified +> 2026-07-09), so `gcloud logging read` sweeps are exact counts, not samples. Cloud +> Monitoring dashboards/alerts are optional (deferred per plan doc); **there is no +> automatic failover** — serverless NEGs have no LB health checks, so a failed gate is +> acted on by a human running the rollback runbook. + +Ramp ladder: `bs-cloud-run` weight **1 → 5 → 25 → 50 → 100**, holding **~24h** per step. +Advance only when every gate below is green for the full hold. + +## 0. Shell variables + +```bash +export PROJECT=policyengine-api +export HOST=api.policyengine.org # post-Stage-8 this resolves to the LB +export SNAP_DIR=docs/migration/urlmap +export BASELINE=docs/migration/baselines/canonical-gae-via-lb.json # from Stage 8 +export STEP_START=$(date -u +%Y-%m-%dT%H:%M:%SZ) # re-export at each weight change +``` + +## 1. Change the weight (per step) + +Follow the ops-doc procedure exactly: export → diff against the last committed snapshot +(fingerprint-only noise expected) → edit the two weights → import → readback → commit a +timestamped snapshot to `$SNAP_DIR`. + +**Then wait ≥10 minutes before any sampling gate.** URL-map changes propagate to GFEs +over ~5–10 min; `describe` shows new weights immediately, but traffic follows late +(observed in Stages 4–5). Sampling early produces false gate failures. + +## 2. Distribution gate (curl only — header is not CORS-exposed) + +`X-PolicyEngine-Backend` has no `Access-Control-Expose-Headers`, so browsers can't read +it; sampling is curl/server-side only. + +```bash +for i in $(seq 1 "$N"); do + curl -sI --max-time 10 "https://$HOST/liveness-check" | tr -d '\r' \ + | awk -F': ' 'tolower($1)=="x-policyengine-backend" {print $2}' +done | sort | uniq -c +``` + +Acceptable `cloud_run` counts (~mean ± 3σ, binomial; every request must carry the header): + +| Step | N | `cloud_run` acceptable | +|---|---|---| +| 1% | 500 | 1–12 | +| 5% | 400 | 7–33 | +| 25% | 200 | 32–68 | +| 50% | 200 | 79–121 | +| 100% | 200 | 200 (zero `app_engine`) | + +## 3. Error gate — log sweeps over the hold window + +Filter on `bs-*` backend names ONLY: this project's `http_load_balancer` logs also +contain `aef-default-*` entries (App Engine's own front end serving direct traffic), +which are not the LB path. + +```bash +# 5xx per backend (run for bs-cloud-run AND bs-app-engine as control) +gcloud logging read 'resource.type="http_load_balancer" + resource.labels.backend_service_name="bs-cloud-run" + httpRequest.status>=500 timestamp>="'$STEP_START'"' \ + --project "$PROJECT" --limit 50 \ + --format 'value(timestamp, httpRequest.status, httpRequest.requestUrl, httpRequest.latency, jsonPayload.statusDetails)' + +# request volume per backend (exact at sampleRate 1.0) +gcloud logging read 'resource.type="http_load_balancer" + resource.labels.backend_service_name="bs-cloud-run" + timestamp>="'$STEP_START'"' \ + --project "$PROJECT" --format 'value(insertId)' | wc -l +``` + +**Gate: zero steady-state 5xx on `bs-cloud-run`.** One documented exception — the +Stage 5 cold-routing artifact: an isolated 504 cluster (a handful of requests, one +instant, latency ≈300s) coinciding with an instance start is queued-on-boot timeout, +not app failure. Confirm the correlation before excusing it: + +```bash +gcloud logging read 'resource.type="cloud_run_revision" + resource.labels.service_name="policyengine-api" + textPayload:"Starting new instance" timestamp>="'$STEP_START'"' \ + --project "$PROJECT" --format 'value(timestamp)' +``` + +Excuse at most one such cluster per hold, only if its timestamps match an instance +start. Any 5xx NOT explained by a boot window, or repeated clusters, fails the gate → +rollback class (a) and investigate. Repeated clusters mean real traffic scales out more +than expected — raise warm capacity (service-level `minScale`, see ops doc) before +retrying the step. + +Also track scale-out frequency (the count from the query above): it is the leading +indicator for the 504 window. Record it in the step log even when gates pass. + +## 4. Latency gate — baseline comparison (per probe, not pooled) + +Run at least twice per hold (after settle, and before advancing): + +```bash +python scripts/capture_migration_baseline.py --base-url "https://$HOST" \ + --repetitions 10 > /tmp/ramp-step-$(date -u +%Y%m%dT%H%M%SZ).json +python scripts/compare_migration_baseline.py "$BASELINE" /tmp/ramp-step-*.json +``` + +**Gate: comparison script exits 0** (per probe: error rate ≤ baseline + 0.1pp, p95 ≤ +baseline × 1.20). The canonical baseline is captured at Stage 8 (GAE-via-LB, same path, +weights 100/0) and committed to `docs/migration/baselines/`. Never compare against a +direct-GAE (non-LB) baseline — path overhead would contaminate the delta. + +## 5. Functional gate — suites and app E2E + +```bash +API_BASE_URL="https://$HOST" python -m pytest \ + tests/integration/test_live_calculate.py \ + tests/integration/test_live_economy.py \ + tests/integration/test_live_budget_window_cache.py -v +``` + +Green required (economy exercises cross-backend job polling at split weights). Known +flake: the budget-window test can fail under backend queue contention — retry it +isolated before treating as a gate failure. Additionally run the policyengine-app-v2 +E2E suite against production per its own repo instructions. + +## 6. Advance, hold, or roll back + +- All gates green for the full ~24h hold → next step on the ladder. +- Gate failure → [`pr4-app-engine-rollback-runbook.md`](pr4-app-engine-rollback-runbook.md) + class (a) (weights back to `app_engine=100`), then diagnose. +- Record every step (weights, `$STEP_START`, gate outputs, scale-out count, anomalies) + in `docs/migration/history/` when the ramp campaign completes. + +## Accepted risk while alerts are deferred + +With Stage 6 item 4 deferred there is no unattended detection: a regression that starts +mid-hold goes unnoticed until the next manual sweep. During holds, run the step-3 5xx +sweep at least every few waking hours. If this cadence is not sustainable, implement +the deferred alerts (three policies, ~$1.50/mo once Google's alerting billing starts, +$0 before then) rather than lengthening the sweep interval. + +## Log-volume caveat (post-100%) + +Full-rate LB request logging at production volume (~26M req/month) may exceed the +logging free tier. After the 100% step stabilizes, revisit: either accept the logging +cost or reduce `logConfig.sampleRate` — but a reduced sample rate means log-derived +counts are no longer exact, and these gates (if still in use) should move to Monitoring +metrics, which are free and always-on. diff --git a/docs/migration/urlmap/20260708T202008Z-lb-api-5050-stage5-test.yaml b/docs/migration/urlmap/20260708T202008Z-lb-api-5050-stage5-test.yaml new file mode 100644 index 000000000..2095ecf66 --- /dev/null +++ b/docs/migration/urlmap/20260708T202008Z-lb-api-5050-stage5-test.yaml @@ -0,0 +1,12 @@ +creationTimestamp: '2026-07-08T11:10:19.533-07:00' +defaultRouteAction: + weightedBackendServices: + - backendService: https://www.googleapis.com/compute/v1/projects/policyengine-api/global/backendServices/bs-app-engine + weight: 50 + - backendService: https://www.googleapis.com/compute/v1/projects/policyengine-api/global/backendServices/bs-cloud-run + weight: 50 +fingerprint: HmNPfwPFsXk= +id: 1171555047161670757 +kind: compute#urlMap +name: lb-api +selfLink: https://www.googleapis.com/compute/v1/projects/policyengine-api/global/urlMaps/lb-api diff --git a/docs/migration/urlmap/20260708T202008Z-lb-api-restored-100-0.yaml b/docs/migration/urlmap/20260708T202008Z-lb-api-restored-100-0.yaml new file mode 100644 index 000000000..a8673bbf2 --- /dev/null +++ b/docs/migration/urlmap/20260708T202008Z-lb-api-restored-100-0.yaml @@ -0,0 +1,12 @@ +creationTimestamp: '2026-07-08T11:10:19.533-07:00' +defaultRouteAction: + weightedBackendServices: + - backendService: https://www.googleapis.com/compute/v1/projects/policyengine-api/global/backendServices/bs-app-engine + weight: 100 + - backendService: https://www.googleapis.com/compute/v1/projects/policyengine-api/global/backendServices/bs-cloud-run + weight: 0 +fingerprint: aJum_j1F_iI= +id: 1171555047161670757 +kind: compute#urlMap +name: lb-api +selfLink: https://www.googleapis.com/compute/v1/projects/policyengine-api/global/urlMaps/lb-api diff --git a/docs/migration/urlmap/20260709T125507Z-lb-api-5050-stage5-rerun.yaml b/docs/migration/urlmap/20260709T125507Z-lb-api-5050-stage5-rerun.yaml new file mode 100644 index 000000000..ae5ed9c33 --- /dev/null +++ b/docs/migration/urlmap/20260709T125507Z-lb-api-5050-stage5-rerun.yaml @@ -0,0 +1,12 @@ +creationTimestamp: '2026-07-08T11:10:19.533-07:00' +defaultRouteAction: + weightedBackendServices: + - backendService: https://www.googleapis.com/compute/v1/projects/policyengine-api/global/backendServices/bs-app-engine + weight: 50 + - backendService: https://www.googleapis.com/compute/v1/projects/policyengine-api/global/backendServices/bs-cloud-run + weight: 50 +fingerprint: QsBhxBnDkxI= +id: 1171555047161670757 +kind: compute#urlMap +name: lb-api +selfLink: https://www.googleapis.com/compute/v1/projects/policyengine-api/global/urlMaps/lb-api diff --git a/docs/migration/urlmap/20260709T131704Z-lb-api-restored-100-0-rerun.yaml b/docs/migration/urlmap/20260709T131704Z-lb-api-restored-100-0-rerun.yaml new file mode 100644 index 000000000..a8673bbf2 --- /dev/null +++ b/docs/migration/urlmap/20260709T131704Z-lb-api-restored-100-0-rerun.yaml @@ -0,0 +1,12 @@ +creationTimestamp: '2026-07-08T11:10:19.533-07:00' +defaultRouteAction: + weightedBackendServices: + - backendService: https://www.googleapis.com/compute/v1/projects/policyengine-api/global/backendServices/bs-app-engine + weight: 100 + - backendService: https://www.googleapis.com/compute/v1/projects/policyengine-api/global/backendServices/bs-cloud-run + weight: 0 +fingerprint: aJum_j1F_iI= +id: 1171555047161670757 +kind: compute#urlMap +name: lb-api +selfLink: https://www.googleapis.com/compute/v1/projects/policyengine-api/global/urlMaps/lb-api diff --git a/scripts/compare_migration_baseline.py b/scripts/compare_migration_baseline.py new file mode 100644 index 000000000..a87bbccac --- /dev/null +++ b/scripts/compare_migration_baseline.py @@ -0,0 +1,176 @@ +"""Compare two capture_migration_baseline.py summaries for ramp gates. + +This script is intentionally not part of normal CI. It takes a baseline +summary JSON and a candidate summary JSON (both produced by +capture_migration_baseline.py) and exits nonzero unless, for every probe in +the baseline, the candidate's error rate is within --error-rate-margin +(default +0.1pp) and its p95 latency is within --p95-ratio (default x1.20). + +Gates are evaluated PER PROBE, never on the pooled summary: pooling mixes +probe populations with very different latency scales, so a shift in mix +masquerades as a regression (or hides one). Simulation-gateway completion +rows are gated as their own pseudo-probe on completion time. +""" + +from __future__ import annotations + +import argparse +import json +from dataclasses import dataclass +from pathlib import Path + +try: + # Shared with the baseline tool so "p95" means the same thing across the + # GAE-vs-Cloud-Run evidence set. + from scripts.capture_migration_baseline import _percentile +except ImportError: # direct execution: python scripts/compare_migration_baseline.py + from capture_migration_baseline import _percentile + +COMPLETION_SUFFIX = " (completion)" + + +@dataclass(frozen=True) +class ProbeStats: + name: str + request_count: int + failure_count: int + error_rate: float + p95_latency_ms: float | None + + +def _stats_by_probe(summary: dict) -> dict[str, ProbeStats]: + """Recompute per-probe stats from a summary's probes array. + + Request rows (completed is None) group by probe name; completion rows + (completed True/False) form a separate pseudo-probe gated on + completion_ms so slow-but-successful completions are still caught. + """ + groups: dict[str, list[dict]] = {} + for row in summary.get("probes", []): + name = row["name"] + if row.get("completed") is not None: + name += COMPLETION_SUFFIX + groups.setdefault(name, []).append(row) + + stats: dict[str, ProbeStats] = {} + for name, rows in groups.items(): + if name.endswith(COMPLETION_SUFFIX): + successes = [ + row["completion_ms"] + for row in rows + if row.get("completed") and row.get("completion_ms") is not None + ] + failure_count = sum(1 for row in rows if not row.get("completed")) + else: + successes = [row["latency_ms"] for row in rows if row.get("ok")] + failure_count = sum(1 for row in rows if not row.get("ok")) + stats[name] = ProbeStats( + name=name, + request_count=len(rows), + failure_count=failure_count, + error_rate=failure_count / len(rows), + p95_latency_ms=_percentile(successes, 0.95), + ) + return stats + + +def compare( + baseline: dict, + candidate: dict, + *, + error_rate_margin: float, + p95_ratio: float, +) -> tuple[bool, list[str]]: + """Return (passed, report_lines) for candidate vs baseline.""" + baseline_stats = _stats_by_probe(baseline) + candidate_stats = _stats_by_probe(candidate) + lines: list[str] = [] + passed = True + + if not baseline_stats: + return False, ["FAIL: baseline contains no probes"] + + for name in sorted(baseline_stats): + base = baseline_stats[name] + cand = candidate_stats.get(name) + if cand is None: + passed = False + lines.append(f"FAIL {name}: probe missing from candidate") + continue + if base.p95_latency_ms is None: + passed = False + lines.append( + f"FAIL {name}: baseline has no successful requests; " + "recapture the baseline" + ) + continue + + problems = [] + error_rate_limit = base.error_rate + error_rate_margin + if cand.error_rate > error_rate_limit + 1e-9: + problems.append( + f"error rate {cand.error_rate:.4f} > " + f"{base.error_rate:.4f} + {error_rate_margin:.4f}" + ) + p95_limit = base.p95_latency_ms * p95_ratio + if cand.p95_latency_ms is None: + problems.append("no successful requests") + elif cand.p95_latency_ms > p95_limit + 1e-9: + problems.append( + f"p95 {cand.p95_latency_ms:.2f}ms > " + f"{base.p95_latency_ms:.2f}ms x {p95_ratio:.2f}" + ) + + if problems: + passed = False + lines.append(f"FAIL {name}: " + "; ".join(problems)) + else: + lines.append( + f"PASS {name}: error rate {cand.error_rate:.4f} " + f"(≤ {error_rate_limit:.4f}), p95 {cand.p95_latency_ms:.2f}ms " + f"(≤ {p95_limit:.2f}ms)" + ) + + for name in sorted(set(candidate_stats) - set(baseline_stats)): + lines.append(f"NOTE {name}: not in baseline; not gated") + + return passed, lines + + +def main(argv: list[str] | None = None) -> int: + parser = argparse.ArgumentParser( + description="Gate a candidate baseline JSON against a reference " + "baseline JSON, per probe." + ) + parser.add_argument("baseline", help="reference summary JSON path") + parser.add_argument("candidate", help="candidate summary JSON path") + parser.add_argument( + "--error-rate-margin", + type=float, + default=0.001, + help="allowed error-rate increase over baseline (default 0.001 = 0.1pp)", + ) + parser.add_argument( + "--p95-ratio", + type=float, + default=1.20, + help="allowed p95 ratio over baseline (default 1.20)", + ) + args = parser.parse_args(argv) + + baseline = json.loads(Path(args.baseline).read_text()) + candidate = json.loads(Path(args.candidate).read_text()) + passed, lines = compare( + baseline, + candidate, + error_rate_margin=args.error_rate_margin, + p95_ratio=args.p95_ratio, + ) + for line in lines: + print(line) + print("RESULT: PASS" if passed else "RESULT: FAIL") + return 0 if passed else 1 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/tests/unit/test_compare_migration_baseline.py b/tests/unit/test_compare_migration_baseline.py new file mode 100644 index 000000000..753fbec64 --- /dev/null +++ b/tests/unit/test_compare_migration_baseline.py @@ -0,0 +1,177 @@ +import json + +from scripts import capture_migration_baseline, compare_migration_baseline + + +def _summary(rows): + return capture_migration_baseline.summarize(rows) + + +def _probe(name, latency_ms, ok=True, **kwargs): + return capture_migration_baseline.ProbeResult( + name=name, + method="GET", + path=f"/{name}", + status_code=200 if ok else 500, + latency_ms=latency_ms, + ok=ok, + **kwargs, + ) + + +def _baseline_summary(): + return _summary( + [ + _probe("liveness", 10.0), + _probe("liveness", 12.0), + _probe("us_metadata", 100.0), + _probe("us_metadata", 120.0), + ] + ) + + +def test_compare_passes_within_margins(): + candidate = _summary( + [ + _probe("liveness", 11.0), + _probe("liveness", 14.0), + _probe("us_metadata", 110.0), + _probe("us_metadata", 130.0), + ] + ) + + passed, lines = compare_migration_baseline.compare( + _baseline_summary(), candidate, error_rate_margin=0.001, p95_ratio=1.20 + ) + + assert passed + assert all(line.startswith("PASS") for line in lines) + + +def test_compare_fails_on_p95_regression_in_one_probe_only(): + candidate = _summary( + [ + _probe("liveness", 11.0), + _probe("liveness", 12.0), + _probe("us_metadata", 100.0), + _probe("us_metadata", 500.0), + ] + ) + + passed, lines = compare_migration_baseline.compare( + _baseline_summary(), candidate, error_rate_margin=0.001, p95_ratio=1.20 + ) + + assert not passed + assert any(line.startswith("FAIL us_metadata") for line in lines) + assert any(line.startswith("PASS liveness") for line in lines) + + +def test_compare_fails_on_error_rate_regression(): + candidate = _summary( + [ + _probe("liveness", 11.0), + _probe("liveness", 12.0, ok=False), + _probe("us_metadata", 110.0), + _probe("us_metadata", 120.0), + ] + ) + + passed, lines = compare_migration_baseline.compare( + _baseline_summary(), candidate, error_rate_margin=0.001, p95_ratio=1.20 + ) + + assert not passed + assert any( + line.startswith("FAIL liveness") and "error rate" in line for line in lines + ) + + +def test_compare_fails_on_probe_missing_from_candidate(): + candidate = _summary([_probe("liveness", 11.0)]) + + passed, lines = compare_migration_baseline.compare( + _baseline_summary(), candidate, error_rate_margin=0.001, p95_ratio=1.20 + ) + + assert not passed + assert any( + line.startswith("FAIL us_metadata") and "missing" in line for line in lines + ) + + +def test_compare_gates_completion_rows_as_pseudo_probe(): + def completion(ms, completed=True): + return _probe( + "simulation_gateway_completion", + 5.0, + completion_ms=ms, + completed=completed, + ) + + baseline = _summary([_probe("liveness", 10.0), completion(1000.0)]) + candidate = _summary([_probe("liveness", 10.0), completion(5000.0)]) + + passed, lines = compare_migration_baseline.compare( + baseline, candidate, error_rate_margin=0.001, p95_ratio=1.20 + ) + + assert not passed + assert any( + line.startswith("FAIL simulation_gateway_completion (completion)") + for line in lines + ) + + +def test_compare_boundary_ratio_passes_exactly_at_limit(): + baseline = _summary([_probe("liveness", 100.0)]) + candidate = _summary([_probe("liveness", 120.0)]) + + passed, _ = compare_migration_baseline.compare( + baseline, candidate, error_rate_margin=0.001, p95_ratio=1.20 + ) + + assert passed + + +def test_compare_fails_on_unusable_baseline(): + baseline = _summary([_probe("liveness", 10.0, ok=False)]) + candidate = _summary([_probe("liveness", 10.0)]) + + passed, lines = compare_migration_baseline.compare( + baseline, candidate, error_rate_margin=0.001, p95_ratio=1.20 + ) + + assert not passed + assert any("recapture the baseline" in line for line in lines) + + +def test_main_exit_codes_and_output(tmp_path, capsys): + baseline_path = tmp_path / "baseline.json" + candidate_path = tmp_path / "candidate.json" + baseline_path.write_text(json.dumps(_baseline_summary())) + candidate_path.write_text(json.dumps(_baseline_summary())) + + exit_code = compare_migration_baseline.main( + [str(baseline_path), str(candidate_path)] + ) + + assert exit_code == 0 + assert "RESULT: PASS" in capsys.readouterr().out + + regression = _summary( + [ + _probe("liveness", 100.0), + _probe("liveness", 100.0), + _probe("us_metadata", 100.0), + _probe("us_metadata", 5000.0), + ] + ) + candidate_path.write_text(json.dumps(regression)) + + exit_code = compare_migration_baseline.main( + [str(baseline_path), str(candidate_path)] + ) + + assert exit_code == 1 + assert "RESULT: FAIL" in capsys.readouterr().out