From 7268586364984a4077dfac72da640d37ee27a80b Mon Sep 17 00:00:00 2001 From: Milos Kotlar Date: Thu, 14 May 2026 13:49:51 +0200 Subject: [PATCH 1/4] [ci-scan] Broaden duplicate detection to catch sibling KBEs Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- .github/workflows/ci-failure-scan.md | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-) diff --git a/.github/workflows/ci-failure-scan.md b/.github/workflows/ci-failure-scan.md index ac8faf5391488e..81ac1bdab17e74 100644 --- a/.github/workflows/ci-failure-scan.md +++ b/.github/workflows/ci-failure-scan.md @@ -196,7 +196,15 @@ For each `(definition_id, phase, queue, stress_mode, signature)` produced by Ste #### Step 4.2 — Search for an existing KBE -`is:issue is:open label:"Known Build Error" in:body ""`. Try variations: full `[FAIL]` line; assertion text; exception class + test name. On hit, record `existing-kbe #` and continue (the walk does not end — a KBE hit changes the final action, not the inspection). +`is:issue is:open label:"Known Build Error" in:body ""`. Try these variations in order, scanning the first ~10 results of each (GitHub best-match ranking can rank a noisier match above the right one): + +1. Full `[FAIL]` line. +2. Assertion text. +3. Exception class + test name. +4. Test class name + `label:"Known Build Error"`, e.g. `SocketBlockingModeTransitionTests label:"Known Build Error"`. +5. Test class name + area label, no KBE filter, e.g. `SocketBlockingModeTransitionTests label:area-System.Net.Sockets`. + +Variations 4 and 5 catch sibling failures filed for the same test class on a different platform or runtime variant (e.g. android-x64 vs iossimulator), plus pre-existing area-team trackers that lack the `Known Build Error` label. If `search_issues` returns a `[Filtered]` marker on any variation, treat it as a likely existing-issue hit and record `skipped: integrity-filtered candidate, needs human review` instead of filing a fresh KBE. On any hit whose title or body references the same test class on any platform, record `existing-kbe #` (or `linked-tracker #` for variation 5 when the hit lacks the KBE label) and continue (the walk does not end; a hit changes the final action, not the inspection). #### Step 4.3 — Search for an area-team tracker (no KBE label) From d5def7d9f378ea65d8eb012373d18d8f2b1f7257 Mon Sep 17 00:00:00 2001 From: Milos Kotlar Date: Fri, 15 May 2026 13:03:40 +0200 Subject: [PATCH 2/4] Limit integrity-filter skip to KBE searches; allow filing fresh KBE when only variation 5 is filtered Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- .github/workflows/ci-failure-scan.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/workflows/ci-failure-scan.md b/.github/workflows/ci-failure-scan.md index 81ac1bdab17e74..0bde15e978621b 100644 --- a/.github/workflows/ci-failure-scan.md +++ b/.github/workflows/ci-failure-scan.md @@ -204,7 +204,7 @@ For each `(definition_id, phase, queue, stress_mode, signature)` produced by Ste 4. Test class name + `label:"Known Build Error"`, e.g. `SocketBlockingModeTransitionTests label:"Known Build Error"`. 5. Test class name + area label, no KBE filter, e.g. `SocketBlockingModeTransitionTests label:area-System.Net.Sockets`. -Variations 4 and 5 catch sibling failures filed for the same test class on a different platform or runtime variant (e.g. android-x64 vs iossimulator), plus pre-existing area-team trackers that lack the `Known Build Error` label. If `search_issues` returns a `[Filtered]` marker on any variation, treat it as a likely existing-issue hit and record `skipped: integrity-filtered candidate, needs human review` instead of filing a fresh KBE. On any hit whose title or body references the same test class on any platform, record `existing-kbe #` (or `linked-tracker #` for variation 5 when the hit lacks the KBE label) and continue (the walk does not end; a hit changes the final action, not the inspection). +Variations 4 and 5 catch sibling failures filed for the same test class on a different platform or runtime variant (e.g. android-x64 vs iossimulator), plus pre-existing area-team trackers that lack the `Known Build Error` label. If `search_issues` returns a `[Filtered]` marker on variations 1-4 (KBE-labeled searches), treat it as a likely existing-KBE hit and record `skipped: integrity-filtered candidate, needs human review` instead of filing a fresh KBE. A `[Filtered]` marker on variation 5 is NOT sufficient to gate filing: per Step 4.3, plain trackers are not KBE substitutes, so continue to file a fresh KBE and record `linked-tracker: integrity-filtered, needs human review` for cross-linking. On any visible hit whose title or body references the same test class on any platform, record `existing-kbe #` (or `linked-tracker #` for variation 5 when the hit lacks the KBE label) and continue (the walk does not end; a hit changes the final action, not the inspection). #### Step 4.3 — Search for an area-team tracker (no KBE label) From cbee9ea2ca4d93550f2bfd70101b20141ff88e21 Mon Sep 17 00:00:00 2001 From: Milos Kotlar Date: Fri, 15 May 2026 13:36:33 +0200 Subject: [PATCH 3/4] [ci-scan] Always emit full 4-key JSON schema in KBE bodies Build Analysis only treats an issue as a tracking KBE when all four keys (ErrorMessage, ErrorPattern, BuildRetry, ExcludeConsoleLog) are present. Recent scanner runs emitted bodies with only 3 keys (e.g. #128141, #128114, #128189) which parsed as valid JSON but silently broke the Tracking linkage on Build Analysis. Update both templates and the 9-check verification list to require the unused match key as an empty string, matching the canonical Arcade reference template. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- .github/workflows/ci-failure-scan.md | 9 +++++++-- 1 file changed, 7 insertions(+), 2 deletions(-) diff --git a/.github/workflows/ci-failure-scan.md b/.github/workflows/ci-failure-scan.md index 0bde15e978621b..ddefe7ca2db8e2 100644 --- a/.github/workflows/ci-failure-scan.md +++ b/.github/workflows/ci-failure-scan.md @@ -324,6 +324,7 @@ Pull request: ", + "ErrorPattern": "", "BuildRetry": false, "ExcludeConsoleLog": false } @@ -356,6 +357,7 @@ Pull request: ```json { + "ErrorMessage": "", "ErrorPattern": "", "BuildRetry": false, "ExcludeConsoleLog": false @@ -371,7 +373,7 @@ Walk all nine before submission. Canonical reference: [`dotnet/arcade-skills/... 2. Exactly ONE fenced JSON block. 3. Opening fence is exactly three backticks + `json`, lowercase, nothing else on the line. 4. Closing fence is exactly three backticks, same length as open. -5. Exactly one of `ErrorMessage` / `ErrorPattern` is present and non-empty. The unused field is DELETED, not set to `""`. +5. **All four keys** (`ErrorMessage`, `ErrorPattern`, `BuildRetry`, `ExcludeConsoleLog`) are present. Exactly one of `ErrorMessage` / `ErrorPattern` is non-empty; the unused one is `""` (empty string), NOT deleted. Build Analysis only treats an issue as a tracking KBE when the full schema is intact — omitting a key silently breaks `Tracking` linkage even though the JSON itself is valid. 6. The signature is NOT a bare identifier. A fully-qualified test name, a stack-frame line, or a bare exception type all appear in `[PASS]` and `[SKIP]` lines for the same test. Applies to BOTH `ErrorMessage` and `ErrorPattern`. 7. Negative-match smoke test against the failure log: @@ -401,7 +403,10 @@ Both `ErrorMessage` and `ErrorPattern` accept arrays — each element matches a "ErrorMessage": [ "", "" - ] + ], + "ErrorPattern": "", + "BuildRetry": false, + "ExcludeConsoleLog": false } ``` From ee0d7b465593bbd83ccb5d82f605ca527cd33364 Mon Sep 17 00:00:00 2001 From: Milos Kotlar Date: Fri, 15 May 2026 14:29:38 +0200 Subject: [PATCH 4/4] [ci-scan] Widen window for persistent failures and read cited code before recommending fix Addresses two defects raised in #127866: 1. Sample window too narrow. When a signature is 100% red within the ~10-build scan window, the true first-seen-in-window likely predates the window. Step 3 now widens the build-list query up to ~40 builds to find the actual first occurrence, and falls back to a body note when the entire widened scan stays red. 2. Recommended fix did not reflect existing code. Branch C now requires reading the cited or modified file at HEAD before drafting a small fix, and explicitly skips the small-fix PR when the recommendation reduces to "do what the cited file already does". Defect 3 (mis-routed area label) is already structurally addressed because Hard rule 3 drops area labels and delegates to the labeler bot; the "Mobile Platform Failure Scanner" workflow has been consolidated into ci-failure-scan since the original issue, so the mobile-routing oddity is moot. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- .github/workflows/ci-failure-scan.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/.github/workflows/ci-failure-scan.md b/.github/workflows/ci-failure-scan.md index ddefe7ca2db8e2..46ba680de35b0b 100644 --- a/.github/workflows/ci-failure-scan.md +++ b/.github/workflows/ci-failure-scan.md @@ -172,6 +172,8 @@ Classification here drives WHERE the agent reads the signature text from. It doe For each (1)/(2)/(3) signature, compute the tuple `(definition_id, work_item_or_phase, queue, stress_mode, [FAIL]-or-compile-error signature)`. Look back ~10 prior completed builds in the same definition for first-seen-in-window timestamp and occurrence count. +If the same signature appears in *every* sampled build (100% failure rate in the ~10-build window), the true first occurrence likely predates the window. Widen the build-list query (`&%24skip=10`, `&%24skip=20`, ...) up to ~40 additional builds and stop as soon as you find a build where the signature is absent (`succeeded`/`partiallySucceeded` without this signature). Report the build immediately after that gap as `First build it occurred` in the KBE body. If you hit the 40-build cap without finding a gap, set `First build it occurred` to the oldest build you scanned and add `Persistent across the entire scanned window; true origin may predate .` as a body note. + #### Data sources - **AzDO REST.** `https://dev.azure.com/dnceng-public/public/_apis/build/...`. Anonymous, no auth. @@ -265,6 +267,8 @@ Build-break KBEs cannot be disabled — there is no test annotation that can ski Small-fix bounds: <= 20 lines, single file, non-API, non-JIT-codegen, non-GC, non-threading, non-security; the failing test (or compile error) verifies the fix. +Before drafting the fix, read every file you intend to cite or modify at HEAD, and any file the failure log points at, to confirm the change is not already present in `main`. If the recommendation reduces to "do what the cited file already does" (header comment, existing target, existing condition), skip Branch C and record `skipped: recommendation already present in source`. + In addition to the Branch B test-disable PR (test failures) or directly against the existing KBE (build breaks), emit a separate `create_pull_request` for the fix on its own branch. Build-break fixes are limited to obvious mechanical changes (typo, missing `#if`, wrong cast, missing `using`). Body cites (a) failing test or compile error as evidence, (b) root cause, (c) why fix is safe, (d) `Linked KBE: #`, (e) "If this lands before #, that PR can be closed." (omit (e) for build-break fixes). After emitting, record the outcome per signature (Step 6).