diff --git a/.github/actions/core-cicd/evergreen-tracks/README.md b/.github/actions/core-cicd/evergreen-tracks/README.md index 7386cc05bdb..5ed3f3cd33e 100644 --- a/.github/actions/core-cicd/evergreen-tracks/README.md +++ b/.github/actions/core-cicd/evergreen-tracks/README.md @@ -8,26 +8,50 @@ tags plus `_tainted` / `_hold` markers). - **`latest`** moves automatically on every GA release cut (the release pipeline calls `promote --tracks latest --apply`). -- **`standard` / `trailing`** move **only when an operator manually dispatches** the - `evergreen-tracks-promote` GitHub Action. There is no cron — a human running the action - at a maintenance-window tag update is the cadence gate, so tracks never re-point - off-window. Automatic (e.g. daily) promotion stays off until the customer-experience - team green-lights it. When run, each track lands on the newest GA older than its age - threshold (`--standard-days` 14, `--trailing-days` 28). - - The dispatch runs in three jobs: `plan` prints the intended moves (dry-run), `gate` - waits on the `evergreen-tracks-apply` environment's required-reviewer gate, and `apply` - runs after approval — nothing moves until a human reviews the plan and approves. - (One-time repo setup: Settings > Environments > `evergreen-tracks-apply` > Required - reviewers.) The dispatch is scoped to `--tracks standard,trailing` — it never moves - `latest` (the release pipeline owns that). `apply` re-derives its plan from live - registry state at run time and **fails if it no longer matches the approved plan** - (e.g. a hold/taint changed, or a release aged past a threshold during a long approval), - so it can never move tags nobody reviewed — just re-dispatch to review the new plan. - Because the plan excludes `latest`, an unattended `latest` move by the release pipeline - mid-approval doesn't trip the drift check. Only `apply` takes the shared - registry-mutation lock, so a pending approval never blocks the release from moving - `latest`. +- **`standard` / `trailing`** advance on a **daily cron at 06:00 ET**, unattended, via the + `evergreen-tracks-promote` GitHub Action (`10:00 UTC` — GitHub cron has no DST, so it's + 05:00 ET in winter). Each track lands on the newest GA older than its age threshold + (`--standard-days` 14, `--trailing-days` 28). Daily promotion does not mean daily tag + movement: the planner is forward-only and age-gated, so a track moves only on the day a + release actually crosses its threshold. + + The same workflow can be **dispatched manually as the break-glass path** — to review a + plan before it lands, or to move tags off-cycle. On that path a `gate` job waits on the + `evergreen-tracks-apply` environment's required-reviewer rule before `apply` runs. The + scheduled path skips `gate` entirely, which is what makes it unattended. (One-time repo + setup: Settings > Environments > `evergreen-tracks-apply` > Required reviewers.) + + Both paths are scoped to `--tracks standard,trailing` — this workflow never moves + `latest` (the release pipeline owns that). `apply` re-derives its plan from live registry + state and **fails if it no longer matches what `plan` produced** (e.g. a hold/taint + changed), so it can never move tags nobody planned; the next morning's run re-plans. + Because the plan excludes `latest`, a `latest` move by the release pipeline never trips + the drift check. Only `apply` takes the shared registry-mutation lock, so a pending + approval on the manual path never blocks the release from moving `latest`. + +### Notifications + +The workflow posts to **#dot-releases**, deliberately only when there's something to say: + +| Outcome | Posts | +|---|---| +| `plan` or `apply` failed | 🚨 yes — tags apply one at a time, so it points at the run log for which actually landed | +| A tag moved — a track advanced, or a held track was reconciled back to its hold marker | 🌲 yes — with what moved where | +| Nothing to do (most days) | nothing | + +A rejected break-glass approval leaves `apply` *skipped* rather than failed, so declining a +plan never pages the channel. Notification failures are `continue-on-error` — Slack being +down never fails a promotion. + +### Why unattended daily promotion is safe + +Moving a floating tag deploys nothing. Every customer manifest in +`dotCMS/infrastructure-as-code` pins an immutable `@sha256:`, and the +in-cluster evergreen-tracks reconciler is the only thing that rewrites those pins — it +resolves the track tag at run time, inside its biweekly on-parity Wednesday maintenance +window. A tag moved on a Tuesday has no effect until that window. Nothing in the clusters +watches the tags continuously (no Argo CD Image Updater / Keel, no floating-tag references, +no `imagePullPolicy: Always` on customer pods). ## Operator procedures diff --git a/.github/actions/core-cicd/evergreen-tracks/RUNBOOK.md b/.github/actions/core-cicd/evergreen-tracks/RUNBOOK.md index 3ca0231764c..d0722ae40bf 100644 --- a/.github/actions/core-cicd/evergreen-tracks/RUNBOOK.md +++ b/.github/actions/core-cicd/evergreen-tracks/RUNBOOK.md @@ -1,8 +1,13 @@ # Evergreen Tracks — Operator Runbook Procedures for the three exception operations: **taint a release**, **hold a track**, and -**hold a single environment**. For the routine "advance standard/trailing" operation see -[README.md](README.md). +**hold a single environment**. Routine promotion needs no operator — `standard`/`trailing` +advance on a daily 06:00 ET cron and `latest` moves on every GA cut; see +[README.md](README.md) for that and for the manual break-glass dispatch. + +Because promotion is unattended, these exception controls are how you intervene: **taint** +before a bad release can be picked up, **hold** to freeze or reverse a track, and the IaC +labels to park one environment. Two systems are involved. Know which one you're touching: @@ -187,7 +192,8 @@ Plus the Argo CD UI (Application = the **customer**, not the env) for Synced / H | I need to… | Where | Action | |---|---|---| -| Advance standard/trailing fleet-wide | `evergreen-tracks-promote` | dispatch → approve gate | +| Advance standard/trailing fleet-wide | `evergreen-tracks-promote` | automatic — daily 06:00 ET cron | +| Advance them off-cycle / review a plan first | `evergreen-tracks-promote` | dispatch → approve gate (break-glass) | | Stop any track landing on a bad release | `evergreen-tracks-admin` | `taint` | | Freeze a track / pull it off a bad release | `evergreen-tracks-admin` | `hold` | | Resume a frozen track | `evergreen-tracks-admin` | `release-hold` | diff --git a/.github/workflows/cicd_evergreen-tracks-promote.yml b/.github/workflows/cicd_evergreen-tracks-promote.yml index 1d5579fd923..f0e81d03b7d 100644 --- a/.github/workflows/cicd_evergreen-tracks-promote.yml +++ b/.github/workflows/cicd_evergreen-tracks-promote.yml @@ -1,19 +1,34 @@ name: evergreen-tracks-promote -# Manual-only: standard/trailing move when an operator dispatches this workflow, -# not on a schedule. A human running this action at a maintenance-window tag -# update IS the cadence gate — customers get a predictable "your next update is -# when we run this" answer without a cron moving tags off-window. No cron until -# the customer-experience team green-lights automatic (e.g. daily) promotion. -# `latest` still moves per-release via cicd_6-release.yml (--tracks latest). +# Advances the floating `standard` / `trailing` tags. `latest` is NOT touched here — +# the release pipeline moves it on every GA cut (cicd_6-release.yml, --tracks latest). # -# Flow: the `plan` job always runs a dry-run and prints the tag moves; the -# `gate` job then waits on the `evergreen-tracks-apply` environment's -# required-reviewer gate; `apply` runs after approval. Nothing moves until a -# human reviews the plan and approves the deployment. (One-time repo setup: -# Settings > Environments > evergreen-tracks-apply > Required reviewers. -# Without reviewers configured the gate passes unattended — the gate is the -# protection rule, not the YAML.) +# Two triggers, one engine: +# * SCHEDULE (daily, 06:00 ET) — runs plan → apply unattended. No human in the loop. +# * MANUAL dispatch — runs plan → gate → apply, where `gate` waits on the +# `evergreen-tracks-apply` required-reviewer environment. This is the +# break-glass path: use it to review a plan before it lands, or to move tags +# off-cycle. (One-time repo setup: Settings > Environments > +# evergreen-tracks-apply > Required reviewers. Without reviewers configured the +# gate passes unattended — the gate is the protection rule, not the YAML.) +# +# Why unattended daily promotion is safe: moving a floating tag does not deploy +# anything. Every customer manifest in dotCMS/infrastructure-as-code pins an +# immutable @sha256:, and the in-cluster evergreen-tracks +# reconciler is the only thing that rewrites those pins — it resolves the track +# tag at run time, inside its biweekly on-parity Wednesday maintenance window. +# A tag moved on a Tuesday therefore has no effect until that window. Nothing in +# the clusters watches the tags continuously (no Argo Image Updater / Keel, no +# floating-tag references, no imagePullPolicy: Always on customer pods). +# +# Note that daily promotion does not mean daily tag movement: the planner is +# forward-only and age-gated (standard >= 14d, trailing >= 28d), so a track moves +# only on the day a release actually crosses its threshold. on: + schedule: + # 10:00 UTC = 06:00 EDT / 05:00 EST. GitHub cron is UTC-only with no DST + # handling, so this drifts an hour in winter. Harmless here — the reconciler's + # maintenance window, not this job's clock, governs when images actually roll. + - cron: '0 10 * * *' workflow_dispatch: inputs: repo: @@ -61,11 +76,13 @@ jobs: id: plan working-directory: .github/actions/core-cicd/evergreen-tracks env: - REPO: ${{ github.event.inputs.repo }} - STANDARD_DAYS: ${{ github.event.inputs.standard_days }} - TRAILING_DAYS: ${{ github.event.inputs.trailing_days }} + # Fallbacks are required: a `schedule` event carries no inputs at all, + # so github.event.inputs.* is empty on the cron path. + REPO: ${{ github.event.inputs.repo || 'dotcms/dotcms' }} + STANDARD_DAYS: ${{ github.event.inputs.standard_days || '14' }} + TRAILING_DAYS: ${{ github.event.inputs.trailing_days || '28' }} run: | - echo "repo=$REPO (dry-run) — review this plan, then approve the apply job" + echo "repo=$REPO (dry-run plan; trigger=${{ github.event_name }})" # --tracks standard,trailing: this workflow never moves `latest` — the # release pipeline owns it and moves it unattended (possibly mid-approval). # Scoping here keeps the drift check below immune to `latest` churn. @@ -82,15 +99,20 @@ jobs: echo 'EVERGREEN_PLAN_EOF' } >> "$GITHUB_OUTPUT" - # Approval gate: carries the evergreen-tracks-apply environment's - # required-reviewer rule and nothing else. Pauses after `plan` until a human - # reviews the plan output and approves. Deliberately a separate job from - # `apply`: GitHub acquires a job's concurrency slot BEFORE evaluating its - # environment protection rules (github.com/orgs/community/discussions/17401), - # so putting the registry lock on this waiting job would block the release - # pipeline's latest-promote for as long as the approval sits pending. + # Approval gate — MANUAL DISPATCH ONLY (the break-glass path). Carries the + # evergreen-tracks-apply environment's required-reviewer rule and nothing else, + # pausing after `plan` until a human reviews the plan output and approves. + # Skipped entirely on the daily schedule, which is what makes the cron + # unattended; `apply` treats a skipped gate as "no approval required". + # + # Deliberately a separate job from `apply`: GitHub acquires a job's concurrency + # slot BEFORE evaluating its environment protection rules + # (github.com/orgs/community/discussions/17401), so putting the registry lock on + # this waiting job would block the release pipeline's latest-promote for as long + # as the approval sits pending. gate: needs: plan + if: github.event_name == 'workflow_dispatch' runs-on: ubuntu-${{ vars.UBUNTU_RUNNER_VERSION || '24.04' }} environment: evergreen-tracks-apply steps: @@ -104,18 +126,32 @@ jobs: # cancel-in-progress is false: queueing is safer than aborting mid-promote # and leaving tags half-moved. # - # `apply` re-derives the plan from live registry state at run time and FAILS - # if it differs from the approved plan (e.g. a hold/taint marker changed, or a - # release aged past a track threshold during a long approval), so it can never - # move tags nobody reviewed. The plan is scoped to standard,trailing, so an - # unattended `latest` move by the release pipeline mid-approval does NOT trip - # it. Drift is rare — when it happens, re-dispatch to review the new plan. + # `apply` re-derives the plan from live registry state at run time and FAILS if + # it differs from the plan job's (e.g. a hold/taint marker changed, or a release + # aged past a track threshold while a human sat on the gate), so it can never + # move tags nobody planned. The plan is scoped to standard,trailing, so an + # unattended `latest` move by the release pipeline does NOT trip it. Drift is + # rare — on the manual path, re-dispatch to review the new plan; on the daily + # cron the next morning's run simply re-plans and applies. + # + # The `if` accepts a SKIPPED gate: that is the scheduled path (gate is + # dispatch-only), and it must not also accept a failed or rejected gate — hence + # the explicit result checks rather than a bare always(). apply: needs: [ plan, gate ] + if: >- + always() + && needs.plan.result == 'success' + && (needs.gate.result == 'success' || needs.gate.result == 'skipped') runs-on: ubuntu-${{ vars.UBUNTU_RUNNER_VERSION || '24.04' }} concurrency: group: evergreen-tracks-registry cancel-in-progress: false + outputs: + # Whether any tag actually moved, and a one-line summary — drives the + # Slack notice so quiet days stay quiet. + moved: ${{ steps.apply.outputs.moved }} + summary: ${{ steps.apply.outputs.summary }} steps: - uses: actions/checkout@v4 - uses: astral-sh/setup-uv@v5 @@ -126,16 +162,18 @@ jobs: username: ${{ secrets.DOCKER_USERNAME }} password: ${{ secrets.DOCKER_TOKEN }} - name: Apply track moves + id: apply working-directory: .github/actions/core-cicd/evergreen-tracks env: - REPO: ${{ github.event.inputs.repo }} - STANDARD_DAYS: ${{ github.event.inputs.standard_days }} - TRAILING_DAYS: ${{ github.event.inputs.trailing_days }} + # Same fallbacks as the plan job — no inputs exist on the schedule path. + REPO: ${{ github.event.inputs.repo || 'dotcms/dotcms' }} + STANDARD_DAYS: ${{ github.event.inputs.standard_days || '14' }} + TRAILING_DAYS: ${{ github.event.inputs.trailing_days || '28' }} APPROVED_PLAN: ${{ needs.plan.outputs.plan }} run: | # Re-derive the plan from CURRENT registry state and confirm it still - # matches what was approved. If a GA landed or a hold/taint changed - # between plan and approval, fail rather than move tags nobody reviewed. + # matches the one the plan job produced. If a hold/taint changed in + # between, fail rather than move tags nobody planned. CURRENT_PLAN=$(uv run evergreen-tracks promote \ --repo "$REPO" \ --tracks standard,trailing \ @@ -144,20 +182,88 @@ jobs: # A never-empty plan ("no track moves needed" at minimum) guards against # a silent pass if plan capture ever breaks (e.g. logging leaves stdout). if [ -z "$APPROVED_PLAN" ] || [ -z "$CURRENT_PLAN" ]; then - echo "::error::Empty plan (approved or current) — cannot verify the approved plan. Re-dispatch." + echo "::error::Empty plan (planned or current) — cannot verify the plan. Re-run." exit 1 fi # sort: the plan is order-insensitive (held-track lines come from a set). if [ "$(printf '%s\n' "$APPROVED_PLAN" | sort)" != "$(printf '%s\n' "$CURRENT_PLAN" | sort)" ]; then - echo "::error::Registry state changed since the plan was approved — refusing to apply. Re-dispatch to review the new plan." - echo "--- approved plan ---"; printf '%s\n' "$APPROVED_PLAN" - echo "--- current plan ---"; printf '%s\n' "$CURRENT_PLAN" + echo "::error::Registry state changed since the plan was computed — refusing to apply. Re-run to act on the new plan." + echo "--- planned ---"; printf '%s\n' "$APPROVED_PLAN" + echo "--- current ---"; printf '%s\n' "$CURRENT_PLAN" exit 1 fi - echo "repo=$REPO — plan unchanged since approval; applying" + echo "repo=$REPO — plan unchanged; applying" uv run evergreen-tracks promote \ --repo "$REPO" \ --tracks standard,trailing \ --standard-days "$STANDARD_DAYS" \ --trailing-days "$TRAILING_DAYS" \ --apply + + # Summarize what moved, for the Slack notice. Most days this is empty + # (the planner is age-gated), and an empty summary posts nothing. + # The optional `(held) ` matches the held-track reconcile line + # ("standard (held) -> reconcile to standard_hold"): the engine really + # does re-point that tag, so it must not be silently omitted here. + MOVES=$(printf '%s\n' "$CURRENT_PLAN" \ + | grep -E '^(standard|trailing) (\(held\) )?-> ' \ + | sed 's/ (sha256:.*//' || true) + if [ -n "$MOVES" ]; then + echo "moved=true" >> "$GITHUB_OUTPUT" + # paste -d takes a cycling CHAR LIST, not a string — join on ';' then + # space it out, so two moves read "a -> x; b -> y". + echo "summary=$(printf '%s' "$MOVES" | paste -sd ';' - | sed 's/;/; /g')" >> "$GITHUB_OUTPUT" + else + echo "moved=false" >> "$GITHUB_OUTPUT" + fi + + # Slack notice to #dot-releases. The cron is unattended, so a silent failure + # would mean tracks quietly stop advancing — this is the only thing that would + # tell anyone. Deliberately quiet on the common case: + # * failure -> post (plan or apply actually errored) + # * moved -> post (a track advanced; newsworthy in a releases channel) + # * no moves -> post NOTHING (most days — the planner is age-gated) + # A rejected break-glass approval leaves `apply` SKIPPED, so a deliberate "no" + # never pages the channel. + # + # Deliberately NOT keyed on `needs.gate.result == 'failure'`: a reviewer + # rejection also reports the gate as `failure` (verified — rejecting marks gate + # failure, apply skipped, run failure), so it is indistinguishable from a gate + # infrastructure failure at the job-result level. Including it would page the + # channel on every intentional rejection. The only thing that would go + # unannounced is an infra failure of the gate job itself — which cannot happen + # on the unattended cron (gate is dispatch-only) and, on the manual path, leaves + # a red run in front of the operator who just triggered it. + notify: + needs: [ plan, gate, apply ] + if: always() + runs-on: ubuntu-${{ vars.UBUNTU_RUNNER_VERSION || '24.04' }} + steps: + - uses: actions/checkout@v4 + - name: Notify failure + if: needs.plan.result == 'failure' || needs.apply.result == 'failure' + continue-on-error: true + uses: ./.github/actions/core-cicd/notification/notify-slack + with: + channel-id: 'CE1TBQU00' # #dot-releases + slack-bot-token: ${{ secrets.SLACK_BOT_TOKEN }} + payload: >- + :rotating_light: *Evergreen track promotion FAILED* + (`${{ github.event.inputs.repo || 'dotcms/dotcms' }}`, trigger `${{ github.event_name }}`). + Tags are applied one at a time, so one track may have moved before the failure — + check the run log for which actually applied. + Customer environments are unaffected until the next in-cluster reconciler window. + <${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}|View run> + + - name: Notify moves + if: needs.apply.result == 'success' && needs.apply.outputs.moved == 'true' + continue-on-error: true + uses: ./.github/actions/core-cicd/notification/notify-slack + with: + channel-id: 'CE1TBQU00' # #dot-releases + slack-bot-token: ${{ secrets.SLACK_BOT_TOKEN }} + payload: >- + :evergreen_tree: *Evergreen track tags moved* on `${{ github.event.inputs.repo || 'dotcms/dotcms' }}` + — ${{ needs.apply.outputs.summary }}. + Environments on these tracks pick this up at their next reconciler maintenance window, + not immediately. <${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}|View run>