feat(devops): add PR CI failure triage bot#41616
Open
Skn0tt wants to merge 3 commits into
Open
Conversation
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
Runs Copilot on a PR's failing tests to judge whether they're caused by the PR or pre-existing flakes, and posts the verdict as a comment.
…n link Prove flakes from the playwright-test-results DuckDB instead of grepping other PRs' report comments, add a 🔴/🟡/🟢 verdict format with a worked example, and inject the agent-run link deterministically in the post job.
Skn0tt
force-pushed
the
skn0tt-ci-failure-triage-bot
branch
from
53580c3 to
5036ab4
Compare
Same footer treatment as the CI triage bot - the post job appends the agent-run link instead of relying on the agent to substitute it.
Contributor
Test results for "MCP"4 failed 7705 passed, 1235 skipped Merge workflow run. |
Contributor
Test results for "tests 1"3 flaky49510 passed, 1164 skipped Merge workflow run. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds a Copilot triage step to
create_test_report.yml. When a PR's test run reports failures, a newtriagejob asks Copilot for a verdict: are these failures likely caused by this PR, or pre-existing flakes / infra noise?It reasons from the PR diff (does it reach the failing test?) and proves flakes from the aggregated CI results in the
playwright-test-resultsDuckDB - it looks for the same test failing on other SHAs/PRs or flipping verdict across runs. No such evidence means it says "uncertain" rather than hand-waving.triageruns untrusted PR code but only holdscopilot-requests: write(no secrets, can't post); a separatepostjob holdsissues: write, runs none of the PR's code, just comments the verdict. Same shape astriage.yml.playwright-devopsskill (pr-ci-triage.md), not baked into the workflow.Also tacks a deterministic agent-run link onto
triage.yml's posted comment, using the same post-job footer pattern.Hand-run against live PRs to tune the flake bar. Two verdicts posted in context so you can see both colours:
path.join()separator bug.