From cd4846d051ed69a3e3027fb9b4ea2139d67deeb1 Mon Sep 17 00:00:00 2001 From: Stephen Freudenthaler Date: Wed, 5 Aug 2026 15:04:13 -0400 Subject: [PATCH 1/4] feat(evergreen-tracks): daily 06:00 ET cron for standard/trailing, gate kept as break-glass MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Promotion is now end-to-end unattended: a daily schedule runs plan -> apply with no human in the loop. Manual dispatch keeps the plan -> gate -> apply flow behind the evergreen-tracks-apply required-reviewer environment as the break-glass path (review a plan before it lands, or move tags off-cycle). - gate is dispatch-only (`if: github.event_name == 'workflow_dispatch'`); apply accepts a SKIPPED gate but not a failed/rejected one, hence explicit result checks rather than a bare always(). - Restore input fallbacks in both jobs: a schedule event carries no inputs, so github.event.inputs.* is empty on the cron path (would have promoted with empty --repo/--standard-days). - Drift check and its messages generalized from "approved" to "planned" — it now guards both paths. - Still scoped --tracks standard,trailing; latest remains the release pipeline's. Safe because moving a floating tag deploys nothing: every customer manifest pins an immutable @sha256:, and the in-cluster reconciler (armed on all four prod clusters as of #8086/#8112) resolves the track tag only at run time inside its biweekly on-parity Wednesday window. Verified no continuous watcher exists (no Argo Image Updater/Keel, no floating-tag refs, no imagePullPolicy: Always on customer pods). Daily promotion also isn't daily tag movement — the planner is forward-only and age-gated (14d/28d). Verified: all six plan/gate result combinations resolve correctly (cron applies, rejected/failed gate does not), 66 tests green. Closes: #36705 Co-Authored-By: Claude Opus 4.8 Claude-Session: https://claude.ai/code/session_01L6UZqGwpjPQrnYyZfYAxiq --- .../core-cicd/evergreen-tracks/README.md | 50 ++++---- .../core-cicd/evergreen-tracks/RUNBOOK.md | 12 +- .../cicd_evergreen-tracks-promote.yml | 112 +++++++++++------- 3 files changed, 111 insertions(+), 63 deletions(-) diff --git a/.github/actions/core-cicd/evergreen-tracks/README.md b/.github/actions/core-cicd/evergreen-tracks/README.md index 7386cc05bdb9..f4196fe3c6bf 100644 --- a/.github/actions/core-cicd/evergreen-tracks/README.md +++ b/.github/actions/core-cicd/evergreen-tracks/README.md @@ -8,26 +8,36 @@ tags plus `_tainted` / `_hold` markers). - **`latest`** moves automatically on every GA release cut (the release pipeline calls `promote --tracks latest --apply`). -- **`standard` / `trailing`** move **only when an operator manually dispatches** the - `evergreen-tracks-promote` GitHub Action. There is no cron — a human running the action - at a maintenance-window tag update is the cadence gate, so tracks never re-point - off-window. Automatic (e.g. daily) promotion stays off until the customer-experience - team green-lights it. When run, each track lands on the newest GA older than its age - threshold (`--standard-days` 14, `--trailing-days` 28). - - The dispatch runs in three jobs: `plan` prints the intended moves (dry-run), `gate` - waits on the `evergreen-tracks-apply` environment's required-reviewer gate, and `apply` - runs after approval — nothing moves until a human reviews the plan and approves. - (One-time repo setup: Settings > Environments > `evergreen-tracks-apply` > Required - reviewers.) The dispatch is scoped to `--tracks standard,trailing` — it never moves - `latest` (the release pipeline owns that). `apply` re-derives its plan from live - registry state at run time and **fails if it no longer matches the approved plan** - (e.g. a hold/taint changed, or a release aged past a threshold during a long approval), - so it can never move tags nobody reviewed — just re-dispatch to review the new plan. - Because the plan excludes `latest`, an unattended `latest` move by the release pipeline - mid-approval doesn't trip the drift check. Only `apply` takes the shared - registry-mutation lock, so a pending approval never blocks the release from moving - `latest`. +- **`standard` / `trailing`** advance on a **daily cron at 06:00 ET**, unattended, via the + `evergreen-tracks-promote` GitHub Action (`10:00 UTC` — GitHub cron has no DST, so it's + 05:00 ET in winter). Each track lands on the newest GA older than its age threshold + (`--standard-days` 14, `--trailing-days` 28). Daily promotion does not mean daily tag + movement: the planner is forward-only and age-gated, so a track moves only on the day a + release actually crosses its threshold. + + The same workflow can be **dispatched manually as the break-glass path** — to review a + plan before it lands, or to move tags off-cycle. On that path a `gate` job waits on the + `evergreen-tracks-apply` environment's required-reviewer rule before `apply` runs. The + scheduled path skips `gate` entirely, which is what makes it unattended. (One-time repo + setup: Settings > Environments > `evergreen-tracks-apply` > Required reviewers.) + + Both paths are scoped to `--tracks standard,trailing` — this workflow never moves + `latest` (the release pipeline owns that). `apply` re-derives its plan from live registry + state and **fails if it no longer matches what `plan` produced** (e.g. a hold/taint + changed), so it can never move tags nobody planned; the next morning's run re-plans. + Because the plan excludes `latest`, a `latest` move by the release pipeline never trips + the drift check. Only `apply` takes the shared registry-mutation lock, so a pending + approval on the manual path never blocks the release from moving `latest`. + +### Why unattended daily promotion is safe + +Moving a floating tag deploys nothing. Every customer manifest in +`dotCMS/infrastructure-as-code` pins an immutable `@sha256:`, and the +in-cluster evergreen-tracks reconciler is the only thing that rewrites those pins — it +resolves the track tag at run time, inside its biweekly on-parity Wednesday maintenance +window. A tag moved on a Tuesday has no effect until that window. Nothing in the clusters +watches the tags continuously (no Argo CD Image Updater / Keel, no floating-tag references, +no `imagePullPolicy: Always` on customer pods). ## Operator procedures diff --git a/.github/actions/core-cicd/evergreen-tracks/RUNBOOK.md b/.github/actions/core-cicd/evergreen-tracks/RUNBOOK.md index 3ca0231764c2..d0722ae40bf5 100644 --- a/.github/actions/core-cicd/evergreen-tracks/RUNBOOK.md +++ b/.github/actions/core-cicd/evergreen-tracks/RUNBOOK.md @@ -1,8 +1,13 @@ # Evergreen Tracks — Operator Runbook Procedures for the three exception operations: **taint a release**, **hold a track**, and -**hold a single environment**. For the routine "advance standard/trailing" operation see -[README.md](README.md). +**hold a single environment**. Routine promotion needs no operator — `standard`/`trailing` +advance on a daily 06:00 ET cron and `latest` moves on every GA cut; see +[README.md](README.md) for that and for the manual break-glass dispatch. + +Because promotion is unattended, these exception controls are how you intervene: **taint** +before a bad release can be picked up, **hold** to freeze or reverse a track, and the IaC +labels to park one environment. Two systems are involved. Know which one you're touching: @@ -187,7 +192,8 @@ Plus the Argo CD UI (Application = the **customer**, not the env) for Synced / H | I need to… | Where | Action | |---|---|---| -| Advance standard/trailing fleet-wide | `evergreen-tracks-promote` | dispatch → approve gate | +| Advance standard/trailing fleet-wide | `evergreen-tracks-promote` | automatic — daily 06:00 ET cron | +| Advance them off-cycle / review a plan first | `evergreen-tracks-promote` | dispatch → approve gate (break-glass) | | Stop any track landing on a bad release | `evergreen-tracks-admin` | `taint` | | Freeze a track / pull it off a bad release | `evergreen-tracks-admin` | `hold` | | Resume a frozen track | `evergreen-tracks-admin` | `release-hold` | diff --git a/.github/workflows/cicd_evergreen-tracks-promote.yml b/.github/workflows/cicd_evergreen-tracks-promote.yml index 1d5579fd923d..2c1319b797ab 100644 --- a/.github/workflows/cicd_evergreen-tracks-promote.yml +++ b/.github/workflows/cicd_evergreen-tracks-promote.yml @@ -1,19 +1,34 @@ name: evergreen-tracks-promote -# Manual-only: standard/trailing move when an operator dispatches this workflow, -# not on a schedule. A human running this action at a maintenance-window tag -# update IS the cadence gate — customers get a predictable "your next update is -# when we run this" answer without a cron moving tags off-window. No cron until -# the customer-experience team green-lights automatic (e.g. daily) promotion. -# `latest` still moves per-release via cicd_6-release.yml (--tracks latest). +# Advances the floating `standard` / `trailing` tags. `latest` is NOT touched here — +# the release pipeline moves it on every GA cut (cicd_6-release.yml, --tracks latest). # -# Flow: the `plan` job always runs a dry-run and prints the tag moves; the -# `gate` job then waits on the `evergreen-tracks-apply` environment's -# required-reviewer gate; `apply` runs after approval. Nothing moves until a -# human reviews the plan and approves the deployment. (One-time repo setup: -# Settings > Environments > evergreen-tracks-apply > Required reviewers. -# Without reviewers configured the gate passes unattended — the gate is the -# protection rule, not the YAML.) +# Two triggers, one engine: +# * SCHEDULE (daily, 06:00 ET) — runs plan → apply unattended. No human in the loop. +# * MANUAL dispatch — runs plan → gate → apply, where `gate` waits on the +# `evergreen-tracks-apply` required-reviewer environment. This is the +# break-glass path: use it to review a plan before it lands, or to move tags +# off-cycle. (One-time repo setup: Settings > Environments > +# evergreen-tracks-apply > Required reviewers. Without reviewers configured the +# gate passes unattended — the gate is the protection rule, not the YAML.) +# +# Why unattended daily promotion is safe: moving a floating tag does not deploy +# anything. Every customer manifest in dotCMS/infrastructure-as-code pins an +# immutable @sha256:, and the in-cluster evergreen-tracks +# reconciler is the only thing that rewrites those pins — it resolves the track +# tag at run time, inside its biweekly on-parity Wednesday maintenance window. +# A tag moved on a Tuesday therefore has no effect until that window. Nothing in +# the clusters watches the tags continuously (no Argo Image Updater / Keel, no +# floating-tag references, no imagePullPolicy: Always on customer pods). +# +# Note that daily promotion does not mean daily tag movement: the planner is +# forward-only and age-gated (standard >= 14d, trailing >= 28d), so a track moves +# only on the day a release actually crosses its threshold. on: + schedule: + # 10:00 UTC = 06:00 EDT / 05:00 EST. GitHub cron is UTC-only with no DST + # handling, so this drifts an hour in winter. Harmless here — the reconciler's + # maintenance window, not this job's clock, governs when images actually roll. + - cron: '0 10 * * *' workflow_dispatch: inputs: repo: @@ -61,11 +76,13 @@ jobs: id: plan working-directory: .github/actions/core-cicd/evergreen-tracks env: - REPO: ${{ github.event.inputs.repo }} - STANDARD_DAYS: ${{ github.event.inputs.standard_days }} - TRAILING_DAYS: ${{ github.event.inputs.trailing_days }} + # Fallbacks are required: a `schedule` event carries no inputs at all, + # so github.event.inputs.* is empty on the cron path. + REPO: ${{ github.event.inputs.repo || 'dotcms/dotcms' }} + STANDARD_DAYS: ${{ github.event.inputs.standard_days || '14' }} + TRAILING_DAYS: ${{ github.event.inputs.trailing_days || '28' }} run: | - echo "repo=$REPO (dry-run) — review this plan, then approve the apply job" + echo "repo=$REPO (dry-run plan; trigger=${{ github.event_name }})" # --tracks standard,trailing: this workflow never moves `latest` — the # release pipeline owns it and moves it unattended (possibly mid-approval). # Scoping here keeps the drift check below immune to `latest` churn. @@ -82,15 +99,20 @@ jobs: echo 'EVERGREEN_PLAN_EOF' } >> "$GITHUB_OUTPUT" - # Approval gate: carries the evergreen-tracks-apply environment's - # required-reviewer rule and nothing else. Pauses after `plan` until a human - # reviews the plan output and approves. Deliberately a separate job from - # `apply`: GitHub acquires a job's concurrency slot BEFORE evaluating its - # environment protection rules (github.com/orgs/community/discussions/17401), - # so putting the registry lock on this waiting job would block the release - # pipeline's latest-promote for as long as the approval sits pending. + # Approval gate — MANUAL DISPATCH ONLY (the break-glass path). Carries the + # evergreen-tracks-apply environment's required-reviewer rule and nothing else, + # pausing after `plan` until a human reviews the plan output and approves. + # Skipped entirely on the daily schedule, which is what makes the cron + # unattended; `apply` treats a skipped gate as "no approval required". + # + # Deliberately a separate job from `apply`: GitHub acquires a job's concurrency + # slot BEFORE evaluating its environment protection rules + # (github.com/orgs/community/discussions/17401), so putting the registry lock on + # this waiting job would block the release pipeline's latest-promote for as long + # as the approval sits pending. gate: needs: plan + if: github.event_name == 'workflow_dispatch' runs-on: ubuntu-${{ vars.UBUNTU_RUNNER_VERSION || '24.04' }} environment: evergreen-tracks-apply steps: @@ -104,14 +126,23 @@ jobs: # cancel-in-progress is false: queueing is safer than aborting mid-promote # and leaving tags half-moved. # - # `apply` re-derives the plan from live registry state at run time and FAILS - # if it differs from the approved plan (e.g. a hold/taint marker changed, or a - # release aged past a track threshold during a long approval), so it can never - # move tags nobody reviewed. The plan is scoped to standard,trailing, so an - # unattended `latest` move by the release pipeline mid-approval does NOT trip - # it. Drift is rare — when it happens, re-dispatch to review the new plan. + # `apply` re-derives the plan from live registry state at run time and FAILS if + # it differs from the plan job's (e.g. a hold/taint marker changed, or a release + # aged past a track threshold while a human sat on the gate), so it can never + # move tags nobody planned. The plan is scoped to standard,trailing, so an + # unattended `latest` move by the release pipeline does NOT trip it. Drift is + # rare — on the manual path, re-dispatch to review the new plan; on the daily + # cron the next morning's run simply re-plans and applies. + # + # The `if` accepts a SKIPPED gate: that is the scheduled path (gate is + # dispatch-only), and it must not also accept a failed or rejected gate — hence + # the explicit result checks rather than a bare always(). apply: needs: [ plan, gate ] + if: >- + always() + && needs.plan.result == 'success' + && (needs.gate.result == 'success' || needs.gate.result == 'skipped') runs-on: ubuntu-${{ vars.UBUNTU_RUNNER_VERSION || '24.04' }} concurrency: group: evergreen-tracks-registry @@ -128,14 +159,15 @@ jobs: - name: Apply track moves working-directory: .github/actions/core-cicd/evergreen-tracks env: - REPO: ${{ github.event.inputs.repo }} - STANDARD_DAYS: ${{ github.event.inputs.standard_days }} - TRAILING_DAYS: ${{ github.event.inputs.trailing_days }} + # Same fallbacks as the plan job — no inputs exist on the schedule path. + REPO: ${{ github.event.inputs.repo || 'dotcms/dotcms' }} + STANDARD_DAYS: ${{ github.event.inputs.standard_days || '14' }} + TRAILING_DAYS: ${{ github.event.inputs.trailing_days || '28' }} APPROVED_PLAN: ${{ needs.plan.outputs.plan }} run: | # Re-derive the plan from CURRENT registry state and confirm it still - # matches what was approved. If a GA landed or a hold/taint changed - # between plan and approval, fail rather than move tags nobody reviewed. + # matches the one the plan job produced. If a hold/taint changed in + # between, fail rather than move tags nobody planned. CURRENT_PLAN=$(uv run evergreen-tracks promote \ --repo "$REPO" \ --tracks standard,trailing \ @@ -144,17 +176,17 @@ jobs: # A never-empty plan ("no track moves needed" at minimum) guards against # a silent pass if plan capture ever breaks (e.g. logging leaves stdout). if [ -z "$APPROVED_PLAN" ] || [ -z "$CURRENT_PLAN" ]; then - echo "::error::Empty plan (approved or current) — cannot verify the approved plan. Re-dispatch." + echo "::error::Empty plan (planned or current) — cannot verify the plan. Re-run." exit 1 fi # sort: the plan is order-insensitive (held-track lines come from a set). if [ "$(printf '%s\n' "$APPROVED_PLAN" | sort)" != "$(printf '%s\n' "$CURRENT_PLAN" | sort)" ]; then - echo "::error::Registry state changed since the plan was approved — refusing to apply. Re-dispatch to review the new plan." - echo "--- approved plan ---"; printf '%s\n' "$APPROVED_PLAN" - echo "--- current plan ---"; printf '%s\n' "$CURRENT_PLAN" + echo "::error::Registry state changed since the plan was computed — refusing to apply. Re-run to act on the new plan." + echo "--- planned ---"; printf '%s\n' "$APPROVED_PLAN" + echo "--- current ---"; printf '%s\n' "$CURRENT_PLAN" exit 1 fi - echo "repo=$REPO — plan unchanged since approval; applying" + echo "repo=$REPO — plan unchanged; applying" uv run evergreen-tracks promote \ --repo "$REPO" \ --tracks standard,trailing \ From 4f584fe8498f87e63936b164c9767084ddde41e7 Mon Sep 17 00:00:00 2001 From: Stephen Freudenthaler Date: Wed, 5 Aug 2026 19:06:52 -0400 Subject: [PATCH 2/4] feat(evergreen-tracks): notify #dot-releases on promotion failure or track move MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The daily cron is unattended, so a silent failure would mean tracks quietly stop advancing with nobody the wiser. Adds a notify job using the existing core-cicd/notification/notify-slack action, posting to #dot-releases (CE1TBQU00). Deliberately quiet: failures post, actual track moves post, and the common "no track moves needed" day posts nothing. A rejected break-glass approval leaves apply SKIPPED rather than failed, so declining a plan never pages the channel. Both steps are continue-on-error — Slack being down never fails a promotion. The apply job now emits moved/summary outputs; the summary join uses `paste -sd ';' | sed` because paste -d takes a cycling char list, not a string. Verified: summary extraction across both-moved / one-moved / no-moves / held-only plans; YAML parses; 66 tests green. Co-Authored-By: Claude Opus 4.8 Claude-Session: https://claude.ai/code/session_01L6UZqGwpjPQrnYyZfYAxiq --- .../core-cicd/evergreen-tracks/README.md | 14 +++++ .../cicd_evergreen-tracks-promote.yml | 61 +++++++++++++++++++ 2 files changed, 75 insertions(+) diff --git a/.github/actions/core-cicd/evergreen-tracks/README.md b/.github/actions/core-cicd/evergreen-tracks/README.md index f4196fe3c6bf..793837a1c188 100644 --- a/.github/actions/core-cicd/evergreen-tracks/README.md +++ b/.github/actions/core-cicd/evergreen-tracks/README.md @@ -29,6 +29,20 @@ tags plus `_tainted` / `_hold` markers). the drift check. Only `apply` takes the shared registry-mutation lock, so a pending approval on the manual path never blocks the release from moving `latest`. +### Notifications + +The workflow posts to **#dot-releases**, deliberately only when there's something to say: + +| Outcome | Posts | +|---|---| +| `plan` or `apply` failed | 🚨 yes — the tags did **not** move (unchanged, not half-moved) | +| A track advanced | 🌲 yes — with what moved where | +| Nothing to do (most days) | nothing | + +A rejected break-glass approval leaves `apply` *skipped* rather than failed, so declining a +plan never pages the channel. Notification failures are `continue-on-error` — Slack being +down never fails a promotion. + ### Why unattended daily promotion is safe Moving a floating tag deploys nothing. Every customer manifest in diff --git a/.github/workflows/cicd_evergreen-tracks-promote.yml b/.github/workflows/cicd_evergreen-tracks-promote.yml index 2c1319b797ab..d36da513211b 100644 --- a/.github/workflows/cicd_evergreen-tracks-promote.yml +++ b/.github/workflows/cicd_evergreen-tracks-promote.yml @@ -147,6 +147,11 @@ jobs: concurrency: group: evergreen-tracks-registry cancel-in-progress: false + outputs: + # Whether any tag actually moved, and a one-line summary — drives the + # Slack notice so quiet days stay quiet. + moved: ${{ steps.apply.outputs.moved }} + summary: ${{ steps.apply.outputs.summary }} steps: - uses: actions/checkout@v4 - uses: astral-sh/setup-uv@v5 @@ -157,6 +162,7 @@ jobs: username: ${{ secrets.DOCKER_USERNAME }} password: ${{ secrets.DOCKER_TOKEN }} - name: Apply track moves + id: apply working-directory: .github/actions/core-cicd/evergreen-tracks env: # Same fallbacks as the plan job — no inputs exist on the schedule path. @@ -193,3 +199,58 @@ jobs: --standard-days "$STANDARD_DAYS" \ --trailing-days "$TRAILING_DAYS" \ --apply + + # Summarize what moved, for the Slack notice. Most days this is empty + # (the planner is age-gated), and an empty summary posts nothing. + MOVES=$(printf '%s\n' "$CURRENT_PLAN" \ + | grep -E '^(standard|trailing) -> ' \ + | sed 's/ (sha256:.*//' || true) + if [ -n "$MOVES" ]; then + echo "moved=true" >> "$GITHUB_OUTPUT" + # paste -d takes a cycling CHAR LIST, not a string — join on ';' then + # space it out, so two moves read "a -> x; b -> y". + echo "summary=$(printf '%s' "$MOVES" | paste -sd ';' - | sed 's/;/; /g')" >> "$GITHUB_OUTPUT" + else + echo "moved=false" >> "$GITHUB_OUTPUT" + fi + + # Slack notice to #dot-releases. The cron is unattended, so a silent failure + # would mean tracks quietly stop advancing — this is the only thing that would + # tell anyone. Deliberately quiet on the common case: + # * failure -> post (plan or apply actually errored) + # * moved -> post (a track advanced; newsworthy in a releases channel) + # * no moves -> post NOTHING (most days — the planner is age-gated) + # A rejected break-glass approval leaves `apply` SKIPPED, not failed, so a + # deliberate "no" never pages the channel. + notify: + needs: [ plan, gate, apply ] + if: always() + runs-on: ubuntu-${{ vars.UBUNTU_RUNNER_VERSION || '24.04' }} + steps: + - uses: actions/checkout@v4 + - name: Notify failure + if: needs.plan.result == 'failure' || needs.apply.result == 'failure' + continue-on-error: true + uses: ./.github/actions/core-cicd/notification/notify-slack + with: + channel-id: 'CE1TBQU00' # #dot-releases + slack-bot-token: ${{ secrets.SLACK_BOT_TOKEN }} + payload: >- + :rotating_light: *Evergreen track promotion FAILED* + (`${{ github.event.inputs.repo || 'dotcms/dotcms' }}`, trigger `${{ github.event_name }}`). + The `standard`/`trailing` tags did NOT move — they are unchanged, not half-moved. + Customer environments are unaffected until the next in-cluster reconciler window. + <${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}|View run> + + - name: Notify moves + if: needs.apply.result == 'success' && needs.apply.outputs.moved == 'true' + continue-on-error: true + uses: ./.github/actions/core-cicd/notification/notify-slack + with: + channel-id: 'CE1TBQU00' # #dot-releases + slack-bot-token: ${{ secrets.SLACK_BOT_TOKEN }} + payload: >- + :evergreen_tree: *Evergreen tracks advanced* on `${{ github.event.inputs.repo || 'dotcms/dotcms' }}` + — ${{ needs.apply.outputs.summary }}. + Environments on these tracks pick this up at their next reconciler maintenance window, + not immediately. <${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}|View run> From 434251da2b70dc6cc5fbdf26fcc0b40c8593875b Mon Sep 17 00:00:00 2001 From: Stephen Freudenthaler Date: Wed, 5 Aug 2026 22:08:18 -0400 Subject: [PATCH 3/4] fix(evergreen-tracks): correct failure-alert claim and catch held-track reconciles MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Two review findings, both confirmed against the engine: 1. The failure alert claimed tags were "unchanged, not half-moved". False: cmd_promote applies moves in a per-track loop and point_tag runs imagetools with check=True, so standard succeeding then trailing failing leaves a real half-moved state — the alert asserted the opposite of what happened. Now points responders at the run log for which tracks actually applied. 2. The Slack summary grep ('^(standard|trailing) -> ') missed the held-track reconcile line ("standard (held) -> reconcile to standard_hold"). The engine physically re-points that tag, so an unattended cron could move a real tag and post nothing. Widened with an optional '(held) ' group. Headline generalized "advanced" -> "tags moved", since a held reconcile is a correction rather than an advance. Verified across five plan shapes: both-advanced, quiet day, held-reconcile-only (previously silent), held+advance, and held-but-consistent (correctly stays quiet — no tag is touched in that case). 66 tests green. Co-Authored-By: Claude Opus 4.8 Claude-Session: https://claude.ai/code/session_01L6UZqGwpjPQrnYyZfYAxiq --- .github/actions/core-cicd/evergreen-tracks/README.md | 4 ++-- .github/workflows/cicd_evergreen-tracks-promote.yml | 10 +++++++--- 2 files changed, 9 insertions(+), 5 deletions(-) diff --git a/.github/actions/core-cicd/evergreen-tracks/README.md b/.github/actions/core-cicd/evergreen-tracks/README.md index 793837a1c188..5ed3f3cd33ea 100644 --- a/.github/actions/core-cicd/evergreen-tracks/README.md +++ b/.github/actions/core-cicd/evergreen-tracks/README.md @@ -35,8 +35,8 @@ The workflow posts to **#dot-releases**, deliberately only when there's somethin | Outcome | Posts | |---|---| -| `plan` or `apply` failed | 🚨 yes — the tags did **not** move (unchanged, not half-moved) | -| A track advanced | 🌲 yes — with what moved where | +| `plan` or `apply` failed | 🚨 yes — tags apply one at a time, so it points at the run log for which actually landed | +| A tag moved — a track advanced, or a held track was reconciled back to its hold marker | 🌲 yes — with what moved where | | Nothing to do (most days) | nothing | A rejected break-glass approval leaves `apply` *skipped* rather than failed, so declining a diff --git a/.github/workflows/cicd_evergreen-tracks-promote.yml b/.github/workflows/cicd_evergreen-tracks-promote.yml index d36da513211b..ad29a349773e 100644 --- a/.github/workflows/cicd_evergreen-tracks-promote.yml +++ b/.github/workflows/cicd_evergreen-tracks-promote.yml @@ -202,8 +202,11 @@ jobs: # Summarize what moved, for the Slack notice. Most days this is empty # (the planner is age-gated), and an empty summary posts nothing. + # The optional `(held) ` matches the held-track reconcile line + # ("standard (held) -> reconcile to standard_hold"): the engine really + # does re-point that tag, so it must not be silently omitted here. MOVES=$(printf '%s\n' "$CURRENT_PLAN" \ - | grep -E '^(standard|trailing) -> ' \ + | grep -E '^(standard|trailing) (\(held\) )?-> ' \ | sed 's/ (sha256:.*//' || true) if [ -n "$MOVES" ]; then echo "moved=true" >> "$GITHUB_OUTPUT" @@ -238,7 +241,8 @@ jobs: payload: >- :rotating_light: *Evergreen track promotion FAILED* (`${{ github.event.inputs.repo || 'dotcms/dotcms' }}`, trigger `${{ github.event_name }}`). - The `standard`/`trailing` tags did NOT move — they are unchanged, not half-moved. + Tags are applied one at a time, so one track may have moved before the failure — + check the run log for which actually applied. Customer environments are unaffected until the next in-cluster reconciler window. <${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}|View run> @@ -250,7 +254,7 @@ jobs: channel-id: 'CE1TBQU00' # #dot-releases slack-bot-token: ${{ secrets.SLACK_BOT_TOKEN }} payload: >- - :evergreen_tree: *Evergreen tracks advanced* on `${{ github.event.inputs.repo || 'dotcms/dotcms' }}` + :evergreen_tree: *Evergreen track tags moved* on `${{ github.event.inputs.repo || 'dotcms/dotcms' }}` — ${{ needs.apply.outputs.summary }}. Environments on these tracks pick this up at their next reconciler maintenance window, not immediately. <${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}|View run> From 131bad7acf66665eb58f7d0db79adbfef6881e64 Mon Sep 17 00:00:00 2001 From: Stephen Freudenthaler Date: Wed, 5 Aug 2026 22:21:08 -0400 Subject: [PATCH 4/4] docs(evergreen-tracks): record why gate failure is excluded from the Slack alert MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Review asked whether a gate-job infrastructure failure should page #dot-releases. Answer: no, and the suggested condition would be harmful. Verified empirically — rejecting a deployment review marks gate=failure, apply=skipped, run=failure, so a rejection and a gate infra failure are indistinguishable from needs.gate.result. Adding `|| needs.gate.result == 'failure'` would therefore page the channel on every deliberate rejection. The only uncovered case is a gate infra failure, which cannot occur on the unattended cron (gate is dispatch-only) and on the manual path leaves a red run in front of the operator who just triggered it. Comment only — no logic change. Co-Authored-By: Claude Opus 4.8 Claude-Session: https://claude.ai/code/session_01L6UZqGwpjPQrnYyZfYAxiq --- .github/workflows/cicd_evergreen-tracks-promote.yml | 13 +++++++++++-- 1 file changed, 11 insertions(+), 2 deletions(-) diff --git a/.github/workflows/cicd_evergreen-tracks-promote.yml b/.github/workflows/cicd_evergreen-tracks-promote.yml index ad29a349773e..f0e81d03b7df 100644 --- a/.github/workflows/cicd_evergreen-tracks-promote.yml +++ b/.github/workflows/cicd_evergreen-tracks-promote.yml @@ -223,8 +223,17 @@ jobs: # * failure -> post (plan or apply actually errored) # * moved -> post (a track advanced; newsworthy in a releases channel) # * no moves -> post NOTHING (most days — the planner is age-gated) - # A rejected break-glass approval leaves `apply` SKIPPED, not failed, so a - # deliberate "no" never pages the channel. + # A rejected break-glass approval leaves `apply` SKIPPED, so a deliberate "no" + # never pages the channel. + # + # Deliberately NOT keyed on `needs.gate.result == 'failure'`: a reviewer + # rejection also reports the gate as `failure` (verified — rejecting marks gate + # failure, apply skipped, run failure), so it is indistinguishable from a gate + # infrastructure failure at the job-result level. Including it would page the + # channel on every intentional rejection. The only thing that would go + # unannounced is an infra failure of the gate job itself — which cannot happen + # on the unattended cron (gate is dispatch-only) and, on the manual path, leaves + # a red run in front of the operator who just triggered it. notify: needs: [ plan, gate, apply ] if: always()