Skip to content

chore(ci): align nightly_deployment mute window with 06:00 UTC deploys#24521

Draft
AztecBot wants to merge 2 commits into
nextfrom
cb/c9a2d4ce1318
Draft

chore(ci): align nightly_deployment mute window with 06:00 UTC deploys#24521
AztecBot wants to merge 2 commits into
nextfrom
cb/c9a2d4ce1318

Conversation

@AztecBot

@AztecBot AztecBot commented Jul 4, 2026

Copy link
Copy Markdown
Collaborator

What

The Grafana nightly_deployment alert mute window (spartan/metrics/grafana/alerts/mutetimes.yaml) was set to 04:00–05:00 UTC, but the nightly deploy workflows all run at 06:00 UTC (cron: "0 6 * * *"):

  • .github/workflows/deploy-next-net.yml (the route that actually applies the nightly_deployment mute in policies.yaml)
  • .github/workflows/deploy-staging-public.yml
  • .github/workflows/deploy-staging-internal.yml

So the deploy-time alert churn was never being muted. This moves the mute window to 06:00–08:30 UTC (2.5h) to cover the deploys.

The 2.5h span accounts for:

  1. GitHub cron not firing exactly on time
  2. the network deploy itself taking a while
  3. the network taking ~3 epochs (~38 min each) to form a committee

Also

Added a concise comment to each nightly-deploy workflow's schedule (and to mutetimes.yaml) noting that changing the deploy timing means updating the Grafana mute window too, so the two don't drift again.

@AztecBot AztecBot added ci-draft Run CI on draft PRs. ci-no-fail-fast Sets NO_FAIL_FAST in the CI so the run is not aborted on the first failure claudebox Owned by claudebox. it can push to this PR. labels Jul 4, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ci-draft Run CI on draft PRs. ci-no-fail-fast Sets NO_FAIL_FAST in the CI so the run is not aborted on the first failure claudebox Owned by claudebox. it can push to this PR.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant