feat: Release Tracks — floating Docker tags (latest/standard/trailing) promotion engine (#36160)

[sfreudenthaler]

Resolves #36160 · Epic #35693

Proposed Changes

• Add a small Python promotion engine at cicd/evergreen-tracks/ (managed with uv) that advances three floating Docker tags — latest / standard / trailing — across the linear GA CalVer stream by release age (newest GA, ~14d, ~28d; thresholds configurable).
• State is registry-only via marker tags: <version>_tainted (forward-only block — a bad release can't propagate to more conservative tracks) and <track>_hold (sticky manual freeze). No separate datastore; the audit trail is the Actions run logs.
• Tags are re-pointed by digest with docker buildx imagetools create (no layer re-push). Age is read from the CalVer date in the version, not build/publish date, so emergency backports of older releases can't be swept into a future promotion.
• Two workflows: cicd_evergreen-tracks-promote.yml (daily cron + dispatch) and cicd_evergreen-tracks-admin.yml (manual taint/hold).
• Document Release Tracks (usage + the "why") in the root README.md.

Checklist

• Tests — 60 unit tests (pure planner, calver, markers, registry parser, CLI); run with cd cicd/evergreen-tracks && uv run pytest.
• Translations — N/A (CI tooling, no UI strings).
• Security Implications Contemplated — see notes below.

Additional Info

Tag control: latest is moved on-demand by the release pipeline (promote-latest job in cicd_6-release.yml, --tracks latest) the moment a GA's images publish, for dotcms/dotcms and dotcms/dotcms-dev — the old deploy-docker latest: true path is unwired (latest: false). The daily cron ages standard/trailing forward and always applies (no separate enable gate). To pause promotion, disable the scheduled workflow or hold the track; to block a bad release, taint it. All registry mutations (release-driven latest, cron, admin) serialize under one concurrency group evergreen-tracks-registry.

Credential scope (important): promotion needs only write, but untaint and release-hold call the Hub delete API — the DOCKER_USERNAME/DOCKER_TOKEN used here must have Read/Write/Delete scope, or those two admin actions fail. (Verified both ways: write-only 403s on delete; RWD succeeds.)

Validation: the full lifecycle (promote-by-age, taint→skip, hold→freeze, release-hold→resume, untaint→restore, teardown) was exercised end-to-end against the dotcms/dotcms-test sandbox and verified by digest. The live smoke can't run in core-workflow-test CI (it intentionally carries no live Docker secrets), so it should run here in core CI / on dispatch.

Security notes: free-text workflow inputs are passed via env: (not interpolated into run:) to avoid expression injection; no tokens are logged; least-privilege permissions: contents: read on the promote workflow.

Out of scope (per epic): LTS-line tracks, Java-variant track tags, the Cloud control-plane UI, and update cadence changes.

🤖 Generated with Claude Code

[chatgpt-codex-connector]

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.

[claude]

Claude finished @sfreudenthaler's task in 55s —— View job

---

Rollback Safety Analysis

• Gather context
• Read rollback-unsafe categories reference
• Analyze PR diff
• Post result and add label

---

Result: ✅ Safe To Rollback

All 21 changed files fall into categories explicitly marked SAFE by the reference:

File(s)	Category Check	Verdict
.github/workflows/cicd_6-release.yml	CI/CD YAML — no DB schema, no ES mapping, no API contract	✅ Safe
.github/workflows/cicd_evergreen-tracks-admin.yml	New CI/CD workflow file	✅ Safe
.github/workflows/cicd_evergreen-tracks-promote.yml	New CI/CD workflow file	✅ Safe
README.md	Documentation only	✅ Safe
cicd/evergreen-tracks/**	Pure Python tooling + tests under cicd/ — no Java, no DB, no ES	✅ Safe

No unsafe categories matched:

• No runonce DB migration tasks (C-1, C-2, C-3, C-4, H-1 through H-7)
• No Elasticsearch mapping changes
• No contentlet_as_json model version bump
• No REST/GraphQL API contract changes (M-3)
• No OSGi interface changes (M-4)
• No VTL viewtool changes (H-8)

The PR is purely additive CI/CD tooling: a Python promotion engine in cicd/evergreen-tracks/, two new GitHub Actions workflows, and documentation updates. Rolling back to the previous release would simply mean the new promotion workflows don't exist — Docker tag promotion would revert to the old deploy-docker latest: true path (note: latest: false in cicd_6-release.yml would need re-evaluation, but that's a CI configuration concern, not a data integrity concern). No application data, database state, or API contracts are affected.

Label added: AI: Safe To Rollback

[github-actions]

🤖 Codex Review — openai.gpt-5.5

**[> [🟠 High] .github/workflows/cicd_evergreen-tracks-admin.yml:34 — DOCKER_USERNAME / DOCKER_TOKEN are set at job scope, so every step/action in the job receives Docker Hub credentials, including third-party setup actions that don’t need them. Keep these secrets scoped only to docker/login-action and the final admin run step.

**[> [🟠 High] .github/workflows/cicd_evergreen-tracks-admin.yml:34 — DOCKER_USERNAME / DOCKER_TOKEN are set at job scope, so every step/action in the job receives Docker Hub credentials, including third-party setup actions that don’t need them. Keep these secrets scoped only to docker/login-action and the final admin run step.

[🟠 High] .github/workflows/cicd_evergreen-tracks-promote.yml:34 — the concurrency group is scoped to ${{ github.workflow }} and ${{ github.ref }}, while the admin workflow has no shared concurrency group. Promote runs from different refs, and promote/admin runs, can still overlap while reading and mutating the same registry tags, which can overwrite holds/taints or move tracks from stale state.

**[> [🟠 High] .github/workflows/cicd_evergreen-tracks-admin.yml:34 — DOCKER_USERNAME / DOCKER_TOKEN are set at job scope, so every step/action in the job receives Docker Hub credentials, including third-party setup actions that don’t need them. Keep these secrets scoped only to docker/login-action and the final admin run step.

[🟠 High] .github/workflows/cicd_evergreen-tracks-promote.yml:34 — the concurrency group is scoped to ${{ github.workflow }} and ${{ github.ref }}, while the admin workflow has no shared concurrency group. Promote runs from different refs, and promote/admin runs, can still overlap while reading and mutating the same registry tags, which can overwrite holds/taints or move tracks from stale state.

[🟡 Medium] cicd/evergreen-tracks/src/evergreen_tracks/cli.py:89 — untaint calls hub_login() before delete_tag(..., apply=args.apply), so even dry-runs require Docker credentials and make a live Docker Hub login request. This breaks the documented dry-run behavior and prevents credential-free validation.

**[> [🟠 High] .github/workflows/cicd_evergreen-tracks-admin.yml:34 — DOCKER_USERNAME / DOCKER_TOKEN are set at job scope, so every step/action in the job receives Docker Hub credentials, including third-party setup actions that don’t need them. Keep these secrets scoped only to docker/login-action and the final admin run step.

[🟠 High] .github/workflows/cicd_evergreen-tracks-promote.yml:34 — the concurrency group is scoped to ${{ github.workflow }} and ${{ github.ref }}, while the admin workflow has no shared concurrency group. Promote runs from different refs, and promote/admin runs, can still overlap while reading and mutating the same registry tags, which can overwrite holds/taints or move tracks from stale state.

[🟡 Medium] cicd/evergreen-tracks/src/evergreen_tracks/cli.py:89 — untaint calls hub_login() before delete_tag(..., apply=args.apply), so even dry-runs require Docker credentials and make a live Docker Hub login request. This breaks the documented dry-run behavior and prevents credential-free validation.

[🟡 Medium] cicd/evergreen-tracks/src/evergreen_tracks/cli.py:114 — release-hold has the same dry-run issue: it logs into Docker Hub before checking/applying the delete, so --apply false still requires credentials and external network access.

---

_{Run: #27556377640 · tokens: in: 26182 · out: 7564 (reasoning: 7230) · total: 33746}

[semgrep-code-dotcms-test]

Legal Risk

certifi 2026.5.20 was released under the MPL-2.0 license, a license that
has been flagged by your organization for consideration.

Recommendation

While merging is not directly blocked, it's best to pause and consider what it means to use this license before continuing. If you are unsure, reach out to your security team or Semgrep admin to address this issue.

[sfreudenthaler]

🟠 High — .github/workflows/cicd_evergreen-tracks-promote.yml:27: the promote workflow has no concurrency group, so a scheduled run and a manual promote/admin run can overlap, both read registry state, then apply stale tag moves.

Addressed in 15f171a. Added a workflow-level concurrency group keyed by github.workflow + github.ref with cancel-in-progress: false, so a scheduled run and a manual promote/admin dispatch are serialized — concurrent runs queue and wait their turn rather than aborting mid-promote and leaving tags half-moved. Queueing (not cancelling) is the safer choice for a tag-promotion engine.

[wezell]

Wait, so :latest is going to become 2 weeks old? Should we maintain latest as is and add another tag called... :current? latest is a well known convention.

[sfreudenthaler]

Wait, so :latest is going to become 2 weeks old? Should we maintain latest as is and add another tag called... :current? latest is a well known convention.

after live discussion decision is to hook our releaser into this as a on-demand invoke that way

• latest moves right away
• only one tool asserts control moving evergreen track tags

[sfreudenthaler]

Wait, so :latest is going to become 2 weeks old? Should we maintain latest as is and add another tag called... :current? latest is a well known convention.

Good catch — latest stays exactly as you'd expect (newest GA), and we keep the convention. Per our live discussion, I wired the releaser into this engine so there's one controller for all three tags instead of two. Pushed in 3539720:

• latest moves on-demand, immediately. A new promote-latest job in cicd_6-release.yml invokes the evergreen-tracks engine right after the release images publish (--tracks latest --apply), for both dotcms/dotcms and dotcms/dotcms-dev. No 24h cron lag.
• Unwired the old path. cicd_6-release.yml no longer asks deploy-docker to move latest (latest: false). The engine is now the single mover of latest/standard/trailing.
• Rollout safety preserved. The release-triggered latest promote applies unconditionally (it replaces always-on behavior); it is not behind EVERGREEN_TRACKS_APPLY, which still gates only the new aged standard/trailing tracks until we flip it on.
• Race closed. All registry mutations (daily cron, admin, and this release-triggered promote) now serialize under one static concurrency group evergreen-tracks-registry — this also addresses the remaining cross-workflow concurrency point from the Codex review.

So standard ≈ 2 weeks and trailing ≈ 4 weeks are the aged tracks; latest is unchanged.

[chatgpt-codex-connector]

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.

[github-actions]

🤖 Bedrock Review — deepseek.v3.2

[🟠 High] .github/workflows/cicd_6-release.yml:154 — The promote-latest job's if condition references github.event.inputs.update_latest, but the deployment job (line 139) already passes latest: false unconditionally. This creates a mismatch: latest is never set to true in deployment, but promote-latest still checks update_latest. This could cause confusion about when latest is actually promoted. The logic should be clarified or unified.

[🟡 Medium] .github/workflows/cicd_evergreen-tracks-admin.yml:70 — The run command constructs $EXTRA by string concatenation (EXTRA="$EXTRA --force"). If FORCE or APPLY contain shell-metacharacters (unlikely from workflow inputs), this could cause injection or unexpected parsing. Safer to use arrays or conditional argument building.

[🟡 Medium] .github/workflows/cicd_evergreen-tracks-promote.yml:70 — Similar shell string concatenation issue with $APPLY. Use explicit conditional argument building.

[🟡 Medium] cicd/evergreen-tracks/src/evergreen_tracks/cli.py:36 — In _current_version, if multiple GA releases share the same digest (e.g., same image tagged with multiple versions), newest(matches) picks the latest by date/build. This is correct, but the scenario should be documented as intentional.

[🟡 Medium] cicd/evergreen-tracks/src/evergreen_tracks/cli.py:84 — The held track reconciliation calls point_tag even when the track digest already matches the hold marker digest (line 81-82 skips). However, point_tag is called for all held tracks in the loop, not just those needing reconciliation. The continue on line 82 prevents the call, so this is fine, but the loop structure could be clearer.

[🟡 Medium] cicd/evergreen-tracks/src/evergreen_tracks/cli.py:108 — _delete_marker requires DOCKER_USERNAME and DOCKER_TOKEN only when apply=True. Dry-run does not need them, which is correct. However, the error message says both "must be set to apply this action" but only checks inside the apply block. This is fine.

[🟡 Medium] cicd/evergreen-tracks/src/evergreen_tracks/executor.py:26 — point_tag uses subprocess.run(check=True) which will raise CalledProcessError on failure. This is appropriate, but the caller (cmd_promote, cmd_admin) does not catch it, so the CLI will exit with a non-zero code and a stack trace. This might be intentional for a CLI tool, but could be made more user-friendly.

[🟡 Medium] cicd/evergreen-tracks/src/evergreen_tracks/executor.py:56 — delete_tag treats HTTP 404 as success (line 55). This is correct for idempotency, but note that other 4xx errors (e.g., 403) would raise. Ensure authentication covers those cases.

[🟡 Medium] cicd/evergreen-tracks/src/evergreen_tracks/registry.py:36 — list_tags assumes resp.json() returns a dict with results and next. If the Hub API changes structure or returns an error, resp.raise_for_status() catches HTTP errors but not malformed JSON. Could add validation for results key.

[🟡 Medium] cicd/evergreen-tracks/tests/test_cli.py:548 — The test file is extensive but only mocks external I/O. However, there is no integration test that actually runs the CLI against a real Docker Hub repo (even a test one). This is acceptable for unit tests, but consider adding a smoke test in CI that runs promote --dry-run on a known test repository to catch regressions in the overall flow.

[🟡 Medium] cicd/evergreen-tracks/tests/test_cli.py:180 — Test test_promote_dry_run_returns_zero asserts point_tag is called with apply=False. However, the test mocks list_tags to return only GA tags, no track tags. This means _current_version returns None for all tracks, so plan may not produce moves, and point_tag might not be called at all. The test's assertion about apply=False may be vacuously true. The test should verify the dry-run behavior more precisely.

[🔴 Critical] cicd/evergreen-tracks/src/evergreen_tracks/cli.py:124 — In cmd_admin for hold action, after creating the hold marker, it also calls point_tag to move the track tag to the same digest (line 131). This is correct. However, if the point_tag for the track fails (e.g., network error), the hold marker is already created, leaving the system in an inconsistent state (track not moved but marked as held). Consider making this atomic or adding a rollback.

[🟠 High] cicd/evergreen-tracks/src/evergreen_tracks/cli.py:115 — In cmd_admin for taint, if the version is not found in digests, it returns error code 2. However, digests only contains tags that have a digest in the registry response. If a version exists but has no digest (unlikely), it will error. This is fine, but the error message could be more specific.

[🟠 High] cicd/evergreen-tracks/src/evergreen_tracks/calver.py:30 — parse_release returns None for invalid GA strings. The function age_days expects a Release object; callers must guard against None. The code appears to do so (e.g., list comprehensions filter None). This is safe but requires careful maintenance.

No Java/Angular code changes were present in the diff, so dotCMS-specific conventions (Config.getStringProperty, Logger, APILocator, etc.) were not applicable.

---

_{Run: #27725151771 · tokens: in: 26139 · out: 1378 · total: 27517}

[sfreudenthaler]

Addressed the automated review feedback. Summary of where things landed:

GPT-5.5 pass (earlier):

• 🟠 High — admin Docker creds at job scope → fixed: DOCKER_USERNAME/DOCKER_TOKEN now scoped to the single admin run step that reads them; docker/login-action already takes them via with: from secrets.
• 🟠 High — promote/admin concurrency race → already addressed: both workflows (and the release pipeline's promote-latest) share the static evergreen-tracks-registry concurrency group, so every registry mutation serializes.
• 🟡 Medium ×2 — untaint/release-hold required creds on dry-run → fixed: the delete path is factored into _delete_marker(), which only logs into Hub when --apply is set; dry-runs need no credentials (covered by new tests).

deepseek.v3.2 pass (latest): no High findings, no criticals. The six "Medium" items are non-issues — four are explicitly self-labeled "No issue / No bug / No gaps" in the review text. The two that read as flags:

• cicd_6-release.yml github.repository == 'dotcms/core' guard — intentional, prevents promote-latest from mutating the registry from forks/renames (same pattern the deployment phase uses).
• admin.yml "duplicate" creds — misread: those are two different steps (docker/login-action via with:, the CLI step via env:); each needs creds in its own form. Scoping to the step is the High-finding fix above.

Also resolved the cicd_6-release.yml merge conflict with main: evergreen-tracks remains the sole controller of the floating latest tag (deployment phase latest: false), and main's new update_latest opt-out is honored on the promote-latest gate for back-patches of older release lines.

60→64 unit tests passing (cd cicd/evergreen-tracks && uv run pytest).

[chatgpt-codex-connector]

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.