Flaky Tests Management: document attempt-to-fix detection across batched commits#37554
Open
TovRudyy wants to merge 4 commits into
Open
Flaky Tests Management: document attempt-to-fix detection across batched commits#37554TovRudyy wants to merge 4 commits into
TovRudyy wants to merge 4 commits into
Conversation
…hed commits Clarify that the attempt-to-fix remediation flow triggers when the unique test key appears in any commit included in the triggering push, not only the most recent commit (batched or squashed pushes, or CI providers that report only the head commit). Note that CI retries of the same push are not double-processed. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Align the docs with the implementation after an impl-vs-docs audit: - State the real 10-commit lookback limit (head commit + recent history), instead of implying every commit in the push is scanned. - Fix the squash & merge wording: only the squash commit's message is read (the original commits no longer exist individually). - Drop the unsupported 'fix is not processed more than once' claim. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Keep the main remediation-flow steps concise; move the 10-commit lookback limit and squash-merge behavior into a new Troubleshooting entry. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
juan-fernandez
approved these changes
Jun 16, 2026
drichards-87
approved these changes
Jun 16, 2026
drichards-87
left a comment
Contributor
There was a problem hiding this comment.
Left some minor feedback from Docs and approved the PR.
| 1. Copy the unique flaky test key that is displayed (for example, `DD_ABC123`). | ||
| 1. Include the test key in your Git commit title or message for the fix (for example, `git commit -m "DD_ABC123"`). | ||
| 1. When Datadog detects the test key in your commit, it automatically triggers the remediation flow for that test: | ||
| 1. When Datadog detects the test key, it automatically triggers the remediation flow for that test. The key does not need to be in the most recent commit, because Datadog also checks the recent commits leading up to it, so the flow still triggers when you push several commits together or when your CI provider reports only the most recent commit. The remediation flow: |
Contributor
There was a problem hiding this comment.
Suggested change
| 1. When Datadog detects the test key, it automatically triggers the remediation flow for that test. The key does not need to be in the most recent commit, because Datadog also checks the recent commits leading up to it, so the flow still triggers when you push several commits together or when your CI provider reports only the most recent commit. The remediation flow: | |
| 1. When Datadog detects the test key, it automatically triggers the remediation flow for that test. The key does not need to be in the most recent commit. Datadog also checks recent preceding commits, so the flow still triggers when you push several commits together or when your CI provider reports only the most recent commit. The remediation flow: |
|
|
||
| ### Attempt-to-fix remediation does not trigger after linking a fix | ||
|
|
||
| After you include the test key (for example, `DD_ABC123`) in a commit, Datadog scans the commit that triggered the test run and up to the 10 most recent commits in its history. If the remediation flow does not start, check the following: |
Contributor
There was a problem hiding this comment.
Suggested change
| After you include the test key (for example, `DD_ABC123`) in a commit, Datadog scans the commit that triggered the test run and up to the 10 most recent commits in its history. If the remediation flow does not start, check the following: | |
| After you include the test key (for example, `DD_ABC123`) in a commit, Datadog scans the commit that triggered the test run and up to the 10 most recent commits before it. If the remediation flow does not start, check the following: |
Comment on lines
+254
to
+256
| - **The key is in an older commit.** Only the 10 most recent commits in the triggering commit's history are scanned. If you push more than 10 commits at once, include the key in the triggering commit or within its 10 most recent commits. | ||
| - **The fix was squash-merged.** A squash merge combines the original commits into a single commit, so Datadog reads only the squash commit's message; the individual pre-squash commits are no longer scanned. Most providers include the original commit messages in the squash commit, so a key in any of them is preserved. If your provider omits them, add the key to the squash commit message. | ||
| - **The key does not match the expected format.** Use the exact key shown in the Flaky Tests Management UI (for example, `DD_ABC123`) in the commit title or message. |
Contributor
There was a problem hiding this comment.
Suggested change
| - **The key is in an older commit.** Only the 10 most recent commits in the triggering commit's history are scanned. If you push more than 10 commits at once, include the key in the triggering commit or within its 10 most recent commits. | |
| - **The fix was squash-merged.** A squash merge combines the original commits into a single commit, so Datadog reads only the squash commit's message; the individual pre-squash commits are no longer scanned. Most providers include the original commit messages in the squash commit, so a key in any of them is preserved. If your provider omits them, add the key to the squash commit message. | |
| - **The key does not match the expected format.** Use the exact key shown in the Flaky Tests Management UI (for example, `DD_ABC123`) in the commit title or message. | |
| - **The key is in an older commit.** Datadog scans only the triggering commit and the 10 most recent commits in its history. If you push more than 10 commits at once, include the key in the triggering commit or one of the 10 most recent commits. | |
| - **The fix was squash-merged.** A squash merge combines the original commits into a single commit, so Datadog reads only the squash commit's message; the individual pre-squash commits are no longer scanned. Most providers include the original commit messages in the squash commit, so a key in any of them is preserved. If your provider omits them, add the key to the squash commit message. | |
| - **The key does not match the expected format.** Use the exact key shown in the Flaky Tests Management UI (for example, `DD_ABC123`) in the commit title or message. |
Contributor
Preview links (active after the
|
Contributor
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR do?
Updates the Flaky Tests Management page to document an improvement to the attempt-to-fix remediation flow: Datadog now detects the unique test key (for example,
DD_ABC123) in any commit included in the triggering push, not only the most recent commit.Do not merge immediately
Motivation
Previously the key was reliably detected only in the commit that triggered the test run (typically the head commit). When commits were pushed together or squashed — or when a CI provider reported only the head commit — the key could be missed if it lived in an earlier commit of the push. The flow now triggers regardless of which commit in the push carries the key, and CI retries of the same push are not double-processed. No user configuration is required.
Scope
One-line content change in
content/en/tests/flaky_management/_index.md, in the Confirm fixes for flaky tests section. User-facing wording only — no API or behavior contract changes.Reference: SDCT-332