Skip to content

Flaky Tests Management: document attempt-to-fix detection across batched commits#37554

Open
TovRudyy wants to merge 4 commits into
masterfrom
olek.rudyy/SDCT-332-ftm-attempt-to-fix-batched-commits
Open

Flaky Tests Management: document attempt-to-fix detection across batched commits#37554
TovRudyy wants to merge 4 commits into
masterfrom
olek.rudyy/SDCT-332-ftm-attempt-to-fix-batched-commits

Conversation

@TovRudyy

@TovRudyy TovRudyy commented Jun 16, 2026

Copy link
Copy Markdown
Contributor

What does this PR do?

Updates the Flaky Tests Management page to document an improvement to the attempt-to-fix remediation flow: Datadog now detects the unique test key (for example, DD_ABC123) in any commit included in the triggering push, not only the most recent commit.

Do not merge immediately

Motivation

Previously the key was reliably detected only in the commit that triggered the test run (typically the head commit). When commits were pushed together or squashed — or when a CI provider reported only the head commit — the key could be missed if it lived in an earlier commit of the push. The flow now triggers regardless of which commit in the push carries the key, and CI retries of the same push are not double-processed. No user configuration is required.

Scope

One-line content change in content/en/tests/flaky_management/_index.md, in the Confirm fixes for flaky tests section. User-facing wording only — no API or behavior contract changes.

Reference: SDCT-332

TovRudyy and others added 3 commits June 16, 2026 16:48
…hed commits

Clarify that the attempt-to-fix remediation flow triggers when the unique test
key appears in any commit included in the triggering push, not only the most
recent commit (batched or squashed pushes, or CI providers that report only the
head commit). Note that CI retries of the same push are not double-processed.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Align the docs with the implementation after an impl-vs-docs audit:
- State the real 10-commit lookback limit (head commit + recent history),
  instead of implying every commit in the push is scanned.
- Fix the squash & merge wording: only the squash commit's message is read
  (the original commits no longer exist individually).
- Drop the unsupported 'fix is not processed more than once' claim.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Keep the main remediation-flow steps concise; move the 10-commit lookback
limit and squash-merge behavior into a new Troubleshooting entry.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@TovRudyy TovRudyy marked this pull request as ready for review June 16, 2026 15:22
@TovRudyy TovRudyy requested review from a team as code owners June 16, 2026 15:22

@drichards-87 drichards-87 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Left some minor feedback from Docs and approved the PR.

1. Copy the unique flaky test key that is displayed (for example, `DD_ABC123`).
1. Include the test key in your Git commit title or message for the fix (for example, `git commit -m "DD_ABC123"`).
1. When Datadog detects the test key in your commit, it automatically triggers the remediation flow for that test:
1. When Datadog detects the test key, it automatically triggers the remediation flow for that test. The key does not need to be in the most recent commit, because Datadog also checks the recent commits leading up to it, so the flow still triggers when you push several commits together or when your CI provider reports only the most recent commit. The remediation flow:

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
1. When Datadog detects the test key, it automatically triggers the remediation flow for that test. The key does not need to be in the most recent commit, because Datadog also checks the recent commits leading up to it, so the flow still triggers when you push several commits together or when your CI provider reports only the most recent commit. The remediation flow:
1. When Datadog detects the test key, it automatically triggers the remediation flow for that test. The key does not need to be in the most recent commit. Datadog also checks recent preceding commits, so the flow still triggers when you push several commits together or when your CI provider reports only the most recent commit. The remediation flow:


### Attempt-to-fix remediation does not trigger after linking a fix

After you include the test key (for example, `DD_ABC123`) in a commit, Datadog scans the commit that triggered the test run and up to the 10 most recent commits in its history. If the remediation flow does not start, check the following:

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
After you include the test key (for example, `DD_ABC123`) in a commit, Datadog scans the commit that triggered the test run and up to the 10 most recent commits in its history. If the remediation flow does not start, check the following:
After you include the test key (for example, `DD_ABC123`) in a commit, Datadog scans the commit that triggered the test run and up to the 10 most recent commits before it. If the remediation flow does not start, check the following:

Comment on lines +254 to +256
- **The key is in an older commit.** Only the 10 most recent commits in the triggering commit's history are scanned. If you push more than 10 commits at once, include the key in the triggering commit or within its 10 most recent commits.
- **The fix was squash-merged.** A squash merge combines the original commits into a single commit, so Datadog reads only the squash commit's message; the individual pre-squash commits are no longer scanned. Most providers include the original commit messages in the squash commit, so a key in any of them is preserved. If your provider omits them, add the key to the squash commit message.
- **The key does not match the expected format.** Use the exact key shown in the Flaky Tests Management UI (for example, `DD_ABC123`) in the commit title or message.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
- **The key is in an older commit.** Only the 10 most recent commits in the triggering commit's history are scanned. If you push more than 10 commits at once, include the key in the triggering commit or within its 10 most recent commits.
- **The fix was squash-merged.** A squash merge combines the original commits into a single commit, so Datadog reads only the squash commit's message; the individual pre-squash commits are no longer scanned. Most providers include the original commit messages in the squash commit, so a key in any of them is preserved. If your provider omits them, add the key to the squash commit message.
- **The key does not match the expected format.** Use the exact key shown in the Flaky Tests Management UI (for example, `DD_ABC123`) in the commit title or message.
- **The key is in an older commit.** Datadog scans only the triggering commit and the 10 most recent commits in its history. If you push more than 10 commits at once, include the key in the triggering commit or one of the 10 most recent commits.
- **The fix was squash-merged.** A squash merge combines the original commits into a single commit, so Datadog reads only the squash commit's message; the individual pre-squash commits are no longer scanned. Most providers include the original commit messages in the squash commit, so a key in any of them is preserved. If your provider omits them, add the key to the squash commit message.
- **The key does not match the expected format.** Use the exact key shown in the Flaky Tests Management UI (for example, `DD_ABC123`) in the commit title or message.

@github-actions

Copy link
Copy Markdown
Contributor

Preview links (active after the build_preview check completes)

Modified Files

@github-actions

Copy link
Copy Markdown
Contributor

⚠️ Possible site support note detected

The following files have a site-region shortcode at the top of the page that mentions a feature being "not available" or "not supported":

  • content/en/tests/flaky_management/_index.md

Are you trying to document site support for a product? If so, the site-region shortcode is not the recommended approach. See the Documenting site support guide for the preferred method.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants