Skip to content

The E2E budget guard counts workflow runs, not Supabase — it cannot see a backend swap #726

Description

@TortoiseWolfe

The E2E circuit breaker does not measure Supabase. It counts GitHub workflow runs and treats
that as a proxy for quota.
As a result it will block a brand-new backend with 100% of its free
tier unused, and it cannot notice that the backend changed at all.

Observed on PR #722, 2026-08-14:

NEXT_PUBLIC_SUPABASE_URL: https://ozbdyopxmeqmwnfsmglp.supabase.co
last 24h ....... 6 / 10
this cycle ..... 30 / 30   (since 2026-08-07)
verdict ........ MONTH_EXCEEDED

The run immediately before it read 29 / 30 ... OK and was green. Nothing changed but the
counter: the green run was the 30th and counted itself into the limit for the next one.

Why the proxy cannot be right

  • scripts/ci/e2e-budget-guard.mjs calls api.github.com/.../workflows/e2e.yml/runs?created>=<since>.
    There is no Supabase API call anywhere in the file.
  • The window starts at a hardcoded BACKEND_EPOCH = '2026-08-07T06:00:00Z' (:66).
    E2E_BUDGET_BACKEND_EPOCH exists as an override (:350-353) but is not plumbed through
    e2e.yml
    , so today only a code edit can move it.
  • The project ref appears in the script exactly once — in a comment. SUPABASE_PROJECT_REF is
    never read, so the counting key is (workflow file, creation time, did-any-shard-run).
    Provisioning a new project cannot reset the counter, and the guard cannot tell that CI is
    now pointed somewhere else.
  • DEFAULT_LIMITS = { day: 10, month: 30 } (:51) are policy numbers derived from "44 runs ≈
    100% of a month", not a measurement.

The #640/#648/#700 hardening is genuinely present and correct — blocked and cancelled runs are
not counted. The count of 30 is real. The question is what it is a count of.

The Aug-7 swap was done correctly — so this is latent, not broken today

Worth stating, because "the counter cannot see the ref" invites the guess that CI is pointed at
a stale project. It is not:

BACKEND_EPOCH                  2026-08-07T06:00:00Z   (e2e-budget-guard.mjs:66)
NEXT_PUBLIC_SUPABASE_URL  set  2026-08-07T06:27:23Z
SUPABASE_PROJECT_REF      set  2026-08-07T06:27:24Z
NEXT_PUBLIC_SUPABASE_ANON_KEY  2026-08-07T06:27:24Z

The epoch was moved to the provisioning time, 27 minutes before the variables were written, and
the local .env points at the same project. ozbdyopxmeqmwnfsmglp is the current backend
and the 30 counted runs are genuine runs against it. Someone did the right thing by hand.

That is exactly the risk: it worked because a human remembered. Nothing links the counter to the
ref, so the next swap silently keeps the old count if the epoch edit is forgotten.

Live trap: three names exist as BOTH a variable and a secret

Found while confirming the above. Same names, different values, different ages:

name variable secret what workflows read
NEXT_PUBLIC_SUPABASE_URL 08-07T06:27 08-07T04:59 vars.* ×12
NEXT_PUBLIC_SUPABASE_ANON_KEY 08-07T06:27 08-07T06:22 vars.* ×8
SUPABASE_PROJECT_REF 08-07T06:27 08-07T04:59 vars.* ×2

Every reference in .github/workflows/ is vars.*. There are zero secrets.* references to
these three.
The secrets are dead, they hold older values than the variables, and updating
one — the obvious move when pointing CI at a new backend — changes nothing and reports no error.
Delete them, or the next swap will look done and not be.

(SUPABASE_KEEPALIVE_REFS names three projects: ozbdyopxmeqmwnfsmglp, vswxgxbjodpgwfgsjrhq,
utxdunkaropkwnrqrsef. Only the first is referenced anywhere else — worth confirming the other
two are still wanted rather than being kept alive out of habit.)

Proposed

  1. Make the project ref part of the counting key, so a backend swap resets the window and a
    forgotten epoch bump cannot silently block a healthy project.
  2. Plumb E2E_BUDGET_BACKEND_EPOCH through e2e.yml so moving the window is config, not a code
    edit.
  3. Read actual usage where possible, so the breaker protects the thing it is named after.
  4. The durable answer is [EPIC] Point E2E at an ephemeral per-runner Supabase — retires the mutex, the 25% cancellation rate, and #567 #575 — local Supabase in CI. If the default PR path runs locally,
    cloud quota stops being the scarce resource and this guard shrinks to covering only the jobs
    that genuinely need the hosted backend.

Also note the gate is total: budgetbuildsmoke/rate-limiting/auth-setupe2e.
A spec that touches no backend at all still cannot run — and chromium-gen carries a
storageState from auth.setup.ts, so it would hit Supabase anyway.

Currently blocking PR #722 and every other PR that needs E2E.

Supersedes the "BACKEND_EPOCH + limit both stale" note.

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingpriority:p2Medium — schedule (feature gaps, partial implementations)

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions