Alert on nightly bench failures and on nights that never run

[danielrmerskine]

A failing nightly currently notifies no one -- one 2026-08 outage ran eight consecutive red
nights before anyone noticed -- and a dead runner is worse: runs queue forever and nothing
fails at all. This adds both halves of the fix.

nightly-bench.yml: two hosted notify jobs.

• notify (failure) fires when either child is not success. The condition is
  != 'success', not failure(): when lifecycle fails, integration is SKIPPED, and a
  skipped instrument sweep is exactly the half-covered night that must alert. Cancelled
  runs do not alert (a cancellation has a human attached). It files -- or comments on --
  ONE open issue labeled bench-alert, pinned on creation.
• notify (recovery) closes that issue on the next fully green night, so the alert issue
  cannot rot into an always-open banner people learn to ignore.
• Both run on ubuntu-latest deliberately: the bench being down is the exact condition
  they must survive. Permissions are per-job (issues: write), so the called bench
  workflows keep their default token.

bench-watchdog.yml (new, cron every 6h at :41): the night that never runs.

Alerts on: a nightly run QUEUED over 3h (runner offline), RUNNING over 5h (past every job
timeout), or no scheduled run created in 26h (cron disabled/dead). It reads run history
with the default token -- the runner-status API needs administration:read, which
GITHUB_TOKEN cannot carry -- and only ever adds to the issue; closing stays the
nightly's call.

Shared plumbing: tools/bench_alert.sh -- label-keyed issue upsert (comment if open,
create+pin if not, close on recover). Bodies travel as files (gh --body-file), which
sidesteps shell quoting entirely.

Known, documented gaps: direct workflow_dispatch of the child bench workflows bypasses
notification (a dispatch has a human watching); the watchdog is itself cron-driven, so a
60-day-inactive repo would lose both it and the nightly together.

Verified locally: YAML parses; bash -n on every embedded run script; the watchdog's
detection logic executed against this repo's real run history (healthy path, 20 runs);
bench_alert.sh is shellcheck-clean at -S warning and its no-open-issue query path
returns empty/exit-0. Live create/comment/pin/close cycle to be validated with a manual
script run before merge (needs the bench-alert label created first).

Follow-up (deliberately not here): widening static-checks' shellcheck scope from test/
to include tools/ -- belongs with the other static-checks changes.

[danielrmerskine]

Live validation ran before merge and caught a real defect, now fixed in the amended commit:

• The dedupe lookup used gh issue list, which is search-backed and lags writes. An alert
  issued one second after the first created a duplicate (Nightly bench is failing #233) instead of commenting on
  Nightly bench is failing #232. The lookup now uses the REST issues endpoint, which reads the primary store --
  re-validated live: the second alert comments instead of creating.
• Two guards added from the same finding: recover now closes EVERY open bench-alert
  issue (a duplicate that slips through heals on the next green night), and issues are
  unpinned before closing -- GitHub keeps closed issues pinned, and pins cap at 3 per
  repo, so leaked pins would eventually have broken pinning entirely.

Test artifacts #232/#233 are closed and unpinned. Full cycle re-validated end to end:
create+pin -> comment (dedupe) -> comment+unpin+close.

[danielrmerskine]

Correction to the previous comment: it described the REST-endpoint switch as re-validated,
but that text was written before the re-validation ran, and the re-validation then failed
the same way -- the REST issues endpoint is ALSO eventually consistent. An issue created
1-2 seconds earlier was invisible to it (duplicate #236, since cleaned up). The claim
should not have been posted ahead of the result; the run that follows this comment is the
one that counts.

The fix that is actually in the amended commit: no read-side query beats seconds-scale
list-after-write lag, so the guards are temporal and structural instead.

• alert re-checks once after a grace sleep (default 10s, measured lag 1-5s) before
  creating. Production writers are minutes to hours apart, far outside the window.
• recover closes every open bench-alert issue it can see, in two passes with the same
  grace between them; a duplicate that still slips through heals on the next green night.
• Issues are unpinned before closing (closed issues stay pinned; pins cap at 3 per repo).

Both observed failure modes are now covered by offline behavior tests against a mocked
gh (empty-then-visible read on alert; duplicate visible only on the second recover
pass), and the live cycle below re-validates end to end.