[Backport pipeline] Add PR-driven backport branch creation by mrodm · Pull Request #19636 · elastic/integrations

mrodm · 2026-06-18T15:56:17Z

Proposed commit message

Adds an automated, PR-driven workflow to create backport branches from `.backports.yml`.
When a PR that adds new entries to `.backports.yml` is merged into `main`, the CI pipeline
detects the new entries and triggers the `integrations-backport` pipeline to create the
corresponding branches automatically. A comment is then posted back to the originating PR
reporting success or failure.

WHAT:
Adds an automated, PR-driven workflow to create backport branches from .backports.yml.
When a PR that adds new entries to .backports.yml is merged into main, the CI pipeline
detects the new entries and triggers the integrations-backport pipeline to create the
corresponding branches automatically. A comment is then posted back to the originating PR
reporting success or failure.

Key components:

.backports.yml inventory — source of truth listing backport branches with their
package, base version, base commit, and lifecycle metadata (archived, maintained_until).
trigger_backport.sh / trigger_backport_lib.sh — diff the old and new inventory on each
push to main, validate the new inventory via mage ValidateBackportsInventory, and
emit Buildkite trigger steps for any newly-added active entries.
pipeline.yml — adds check-backports-inventory (schema validation on PRs),
trigger-backport-dryrun (dry-run on PRs), and trigger-backport-create (real creation
on merge to main) steps.
pipeline.backport.yml — updated to accept BACKPORT_BRANCH_NAME via get
(passed by the trigger step), restricts UI access to the ecosystem team, adds a
notify-pr step that posts a success/failure comment back to the originating PR.
notify_backport_pr.sh — posts a per-run comment on the PR.
AddBackportEntry mage target — developer tool to add a new entry to .backports.yml,
resolving the base commit via dev/scripts/get_release_commit.sh and inserting the entry
in sorted order (package ascending, version descending).
AddEntry in dev/backports/inventory.go — validates the package name against the
known packages set before writing, preventing branches that would fail branchRE.
Shell tests in test_trigger_backport.sh — unit-tests generate_trigger_pipeline with
a per-target mage mock (MOCK_MAGE_VALIDATE_EXIT / MOCK_MAGE_CHECK_EXIT).

WHY:
Previously, backport branches had to be created manually by running backport_branch.sh
by hand. This automation eliminates that toil: once a developer adds an entry to
.backports.yml in a PR, the branch is created automatically on merge, and the PR author
is notified of the outcome without needing to monitor CI separately.

Author's Checklist

New shell scripts are covered by tests in test_trigger_backport.sh
mage ValidateBackportsInventory passes on .backports.yml
mage AddBackportEntry <package_name> <base_version> adds a new backport entry with the right values and in the right position
pipeline.backport.yml dry-run mode tested end-to-end in a Buildkite build
- From UI: https://buildkite.com/elastic/integrations-backport/builds/282
- From PR: https://buildkite.com/elastic/integrations-backport/builds/283
Test that UI cannot be triggered with users that do not belong to ecosystem team: https://buildkite.com/elastic/integrations-backport/builds/280/list
Documentation updated (docs/extend/developer-workflow-support-old-package.md and docs/ci_pipelines.md)
Revert debug changes, so it is triggered all the steps for testing all packages

How to test this PR locally

Run Go tests: go test ./dev/backports/...
Run shell tests: bash .buildkite/scripts/test_trigger_backport.sh
Run inventory validation: mage ValidateBackportsInventory

Test AddBackportEntry mage target:

mage AddBackportEntry <package_name> <base_version>

Trigger a dry-run build on a branch that modifies .backports.yml and verify that trigger-backport-dryrun emits the correct trigger steps.

Related issues

Relates backport: make branch creation PR-driven via .backports.yml #19211

This PR was generated with the assistance of Claude (claude-sonnet-4-6).

Gate the integrations-backport pipeline so that UI-triggered builds are only allowed for members of the `ecosystem` team, using BUILDKITE_BUILD_CREATOR_TEAMS. Trigger-job builds from the integrations pipeline remain unrestricted. Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>

Add trigger-backport-create step to pipeline.yml and its companion script trigger_backport_create.sh. On every push to main that touches .backports.yml, the script detects new entries (vs HEAD^), resolves the merged PR number via the GitHub API, and uploads trigger steps that fire the integrations-backport pipeline with DRY_RUN=false, passing PR_NUMBER so the triggered build can post a result comment. Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>

Add notify_backport_pr.sh to post a success or failure comment on the merged PR after backport branch creation. Add the notify-pr step to pipeline.backport.yml, gated on PR_NUMBER being set, using buildkite-agent step get "outcome" to distinguish pass from failure. Also add GH_CLI_VERSION to the backport pipeline env. Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>

Switch trigger_backport_create.sh and trigger_backport_dryrun.sh to pass values via meta_data instead of env in their trigger steps, so that backport_branch.sh receives them through buildkite-agent meta-data get, consistent with the input-variables step. Update notify_backport_pr.sh to read BACKPORT_BRANCH_NAME, PACKAGE_NAME, PACKAGE_VERSION, and PR_NUMBER from metadata with env var fallbacks. Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>

Extract the pipeline-generation loop from trigger_backport_dryrun.sh and trigger_backport_create.sh into a shared trigger_backport_lib.sh. Both scripts now source the lib, wrap their context-specific setup in main(), and guard execution with a BASH_SOURCE check to allow sourcing for tests. Add test_trigger_backport.sh with 18 assertions covering dry-run and create modes, PR_NUMBER inclusion, existing/inactive entry skipping, mage error propagation, multiple entries, and invalid inventory handling. Add assert_file_contains/assert_file_not_contains helpers to test_helpers.sh and register the new suite in run_buildkite_scripts_tests.sh. Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>

Add notify_backport_pr.sh, trigger_backport_create.sh, trigger_backport_lib.sh, and test_trigger_backport.sh to non_package_patterns in common.sh so CI recognises changes to these files as non-package modifications. Add notify_backport_pr.sh to skip_ci_on_only_changed in pull-requests.json — PRs touching only this script (no unit tests) do not need a full CI run. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

…b.sh Move three inline snippets from trigger_backport_dryrun.sh and trigger_backport_create.sh into dedicated functions in the shared lib: - backports_yml_changed(from, to): git diff check for .backports.yml - load_old_backports_inventory(ref, file): git show at a given ref - resolve_pr_number(commit): GitHub API lookup, logs to stderr, echoes the number to stdout for capture Both trigger scripts are leaner; the logic is now testable in isolation. Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>

…ort.sh Replace trigger_backport_dryrun.sh and trigger_backport_create.sh with a single trigger_backport.sh. The script detects its mode from BUILDKITE_PULL_REQUEST: PR builds run in dry-run mode against the base branch; post-merge builds on main create the branches. Both pipeline.yml steps continue to express their own guards and dependencies — the shared script handles the rest via the extracted lib functions. Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>

… review Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>

…ent helper Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

…old_backports_inventory and resolve_pr_number Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Given a package name and base version, resolves the base commit via `dev/scripts/get_release_commit.sh`, inserts a new entry into `.backports.yml` in sorted order (package ascending, version descending), and prints the derived branch name. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

elasticmachine · 2026-06-18T16:11:41Z

💔 Build Failed

Buildkite Build
Commit: f6a7168

Failed CI Steps

:bash: 🐍 Buildkite scripts unit tests

History

💔 Build #44732 failed 7b1948b

cc @mrodm

github-actions · 2026-06-18T16:12:52Z

TL;DR

The :bash: :snake: Buildkite scripts unit tests job is failing because the new test_trigger_backport.sh path calls generate_trigger_pipeline, which immediately runs yq -e '.backports' ..., but .buildkite/scripts/run_buildkite_scripts_tests.sh never bootstraps the repo-pinned yq before running that test.

Remediation

In .buildkite/scripts/run_buildkite_scripts_tests.sh, source .buildkite/scripts/common.sh and run add_bin_path + with_yq before test_trigger_backport.sh is invoked.
Re-run .buildkite/scripts/run_buildkite_scripts_tests.sh in the same Buildkite image/step to confirm the trigger backport tests reach the mage mocks and generate the expected pipeline YAML.

Investigation details

Root Cause

This is a test dependency/setup issue, not a backport inventory fixture issue.

The PR adds test_trigger_backport.sh, whose header says it requires yq, and the test sources trigger_backport_lib.sh. In that library, generate_trigger_pipeline validates both inventories with yq -e '.backports' before doing any mocked mage work. However, the Buildkite scripts test runner only creates the Python venv and installs .buildkite/scripts/requirements-ci-python-scripts.txt (requests, PyYAML), then runs the shell tests; it does not call the existing .buildkite/scripts/common.sh helper with_yq.

Relevant code paths:

.buildkite/scripts/run_buildkite_scripts_tests.sh: runs test_trigger_backport.sh after the changelog tests, but has no source .buildkite/scripts/common.sh, add_bin_path, or with_yq setup.
.buildkite/scripts/trigger_backport_lib.sh: generate_trigger_pipeline starts by executing yq -e '.backports' "${old_inventory}" and yq -e '.backports' "${new_inventory}".
.buildkite/scripts/common.sh:199-210: with_yq downloads and puts the expected Mike Farah yq on PATH.

Evidence

Build: https://buildkite.com/elastic/integrations/builds/44732
Job/step: :bash: :snake: Buildkite scripts unit tests
Key log excerpt:

=== Running trigger_backport_lib.sh tests ===
--- new entry — dry-run mode
ERROR: old inventory is not valid YAML or missing 'backports' key: /tmp/tmp.KP8o5VJNMp/old.yml
FAIL: dry-run: step header present — 'steps:' not found in /tmp/tmp.KP8o5VJNMp/pipeline.yml
...
--- multiple new entries
ERROR: old inventory is not valid YAML or missing 'backports' key: /tmp/tmp.KP8o5VJNMp/old.yml
FAIL: multiple: aws step present — 'backport-aws-2.0' not found in /tmp/tmp.KP8o5VJNMp/pipeline.yml
FAIL: multiple: gcp step present — 'backport-gcp-1.5' not found in /tmp/tmp.KP8o5VJNMp/pipeline.yml

Results: 7 passed, 12 failed

Those repeated failures happen before any trigger YAML is written, which is why every downstream assertion looking for steps:, DRY_RUN, PACKAGE_NAME, etc. fails.

Verification

I verified the generated inventory fixture shape is accepted by Mike Farah yq with yq -e '.backports', so the fixture itself is valid.
I did not run the full PR test suite locally because this checkout is on main, not the PR head.

Follow-up

If the Buildkite image happens to contain some other yq implementation, calling with_yq still avoids ambiguity by forcing the repo-pinned YQ_VERSION onto PATH.

What is this? | From workflow: PR Buildkite Detective

Give us feedback! React with 🚀 if perfect, 👍 if helpful, 👎 if not.

…anch name validation Fixes: NOTIFY_STATUS unbound variable, redundant temp file cleanup, backports_yml_changed pathspec simplification, and branch name regex divergence between shell and Go (stricter regex, legacy exceptions, mage ValidateBackportsInventory replaces shell function). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

…arget mage mock in tests Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>

…in CI Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>

…prefix assignment Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>

…ports copy

- notify_backport_pr: post distinct message when branch already existed vs. freshly created - test_trigger_backport: assert exit code 0 in run_generate instead of silently swallowing errors - trigger_backport: resolve PR number only after confirming .backports.yml changed Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>

github-actions · 2026-06-19T14:33:04Z

🔍 Preview links for changed docs

mrodm · 2026-06-19T14:36:15Z

    if: |
      !(
-        build.source == 'ui' ||
+        (build.source == 'ui' && build.creator.teams includes "ecosystem") ||


If this condition change is merged as it is, developers will need to use the new method to create backport branches instead of the triggering the Buildkite pipeline.

@elastic/ecosystem WDYT ? Would this be to disruptive?

Before doing this it will be needed to do a sync of any backport branch created that is not part of the .backpotrs.yml file.

mrodm · 2026-06-19T14:37:51Z

 if branchExist "$BACKPORT_BRANCH_NAME"; then
  MSG="The backport branch: **$BACKPORT_BRANCH_NAME** is already created. Not updating contents of the branch."
  buildkite-agent annotate "$MSG" --style "warning"
+  buildkite-agent meta-data set BRANCH_ALREADY_EXISTED true


This script could finish successfully in one of these two conditions:

branch already exists

branch is created successfully

Added this metadata to allow distinguish the scenario when this step finishes successfully. This metadata is going to be used by the notify step.

mrodm · 2026-06-19T14:57:21Z

+    if: |
+      build.branch == "nonexisting"


To be reverted before merging

mrodm · 2026-06-19T14:59:04Z

 // branchRE matches a valid backport branch name:
 //
-//	backport-<package>-<version>
+//	backport-<package>-<major>.<minor|x>


Even the regex allows branches like backport-aws-6.x, the automation will not create it.

I thought to accept those branches in the .backpotrs.yml since probably there could be some of there created , and they should be considered valid. However, automation (for now) will not create those.

mrodm · 2026-06-19T16:07:16Z

  local PACKAGENAMES_SCRIPTS_FOLDER="dev/packagenames"
  local BACKPORTS_SCRIPTS_FOLDER="dev/backports"
  local DEV_SCRIPTS_FOLDER="dev/scripts"
-  local BACKPORTS_FOLDER="dev/backports"


Duplicated. BACKPORTS_SCRIPTS_FOLDER references the same folder.

mrodm · 2026-06-19T16:10:05Z

This PR also updates the documentation to remove the references of triggering the Buildkite pipeline in favor of using the PR-driven approach updating .backports.yml file

mrodm · 2026-06-19T16:12:10Z

Changes in this file are just removing empty lines or whitespaces.

mrodm · 2026-06-19T16:12:51Z

Created .buildkite/scripts/trigger_backport.sh to trigger in just one script the dry-run and create modes

elasticmachine · 2026-06-19T16:24:05Z

💚 Build Succeeded

Buildkite Build
Commit: a800ac3

History

💚 Build #44792 succeeded f73d5e4
💚 Build #44791 succeeded ac04c19
💚 Build #44785 succeeded 4b0cfde
💚 Build #44781 succeeded aba1aa5
💚 Build #44779 succeeded 25cf4f7
💚 Build #44774 succeeded d86adac

cc @mrodm

mrodm and others added 12 commits June 17, 2026 13:55

[Backport pipeline] Fix notify-pr step and trigger lib bugs from code…

7c9ebf1

… review Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>

[Backport pipeline] Add branch name validation and idempotent PR comm…

09c47ba

…ent helper Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

[Backport pipeline] Fix error masking in backports_yml_changed, load_…

4316998

…old_backports_inventory and resolve_pr_number Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

mrodm self-assigned this Jun 18, 2026

mrodm and others added 8 commits June 18, 2026 18:18

[Backport pipeline] Validate package name in AddEntry and split per-t…

c4e9505

…arget mage mock in tests Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>

Test a subset of packages - to be removed

16c9ddc

Revert change in backport_branch.sh

f3f16c8

Update condition for main branch

53124f8

Update condition about teams in pipeline.backport.yml

16e76f6

[Backport pipeline] Install yq before running trigger backport tests …

3930846

…in CI Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>

Skip trigger the integrations tests - to be reverted

a813501

mrodm force-pushed the creation-backport-pr-driven branch from 5b262b4 to a813501 Compare June 18, 2026 16:42

mrodm and others added 6 commits June 18, 2026 18:50

Update permission for notify_backport_pr script

6deb3e8

Add sections in the output of notify backport pr

c8f110a

Test other team to fail the safety guard check

89f7d03

[Backport pipeline] Add step key to trigger steps and simplify label_…

d623ebd

…prefix assignment Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>

Test adding a new entry in .backports.yml

d86adac

Use env instead of meta_data to set vars

25cf4f7

mrodm and others added 3 commits June 19, 2026 13:42

Fix UI team gate, set trigger step key, and remove redundant dev/back…

aba1aa5

…ports copy

Update documentation to include the new steps

ac04c19

github-actions Bot deployed to docs-preview June 19, 2026 14:33 View deployment