Skip to content

Rectify: Marker Substring Match False Positives — Centralized Code-Region-Aware Detection - #4408

Merged
Trecek merged 4 commits into
developfrom
investigation-complete-marker-is-substring-matched-over-the/4386-2
Jul 29, 2026
Merged

Rectify: Marker Substring Match False Positives — Centralized Code-Region-Aware Detection#4408
Trecek merged 4 commits into
developfrom
investigation-complete-marker-is-substring-matched-over-the/4386-2

Conversation

@Trecek

@Trecek Trecek commented Jul 29, 2026

Copy link
Copy Markdown
Collaborator

Summary

INVESTIGATION_COMPLETE_MARKER and REVIEW_APPROACH_MARKER are detected via naive str.__contains__ against raw, unsanitized GitHub issue bodies at two duplicated call sites. This produces false positives when the marker text appears inside markdown fenced code blocks or inline code spans. A production-grade strip_markdown_code_regions utility does not exist in src/ — only a test-only version in tests/_helpers.py. Additionally, the build-execution-map SKILL.md propagates the same vulnerability in prose by instructing the LLM to perform a "simple substring check."

The architectural weakness is twofold: (1) no centralized marker-detection predicate exists — each call site reimplements raw in checks independently, and (2) there is no shared code-region-stripping utility in production code. The fix promotes the stripping utility into IL-0 core/, creates a centralized detect_body_marker() predicate that strips before matching, replaces both duplicated call sites, fixes the SKILL.md prose, and adds false-positive rejection tests for both markers.

Implementation Plan

Plan file: /home/talon/projects/autoskillit-runs/remediation-20260728-142843-376918/.autoskillit/temp/rectify/rectify_marker_substring_match_2026-07-28_223749.md

Closes #4386

🤖 Generated with Claude Code via AutoSkillit


Audit gate bypassed — #4406

audit_impl did not gate this PR. It crashed with
AuditCycleVerificationError: authority is not strict canonical versioned JSON — that is
#4406: audit-impl writes authority.json pretty-printed, while its own verifier requires
the strict canonical profile (sort_keys=True, separators=(",", ":")). The crash is in the
cycle-artifact machinery after the audit itself completed, so it is not a quality finding
against this change.

The bypass was explicitly authorized by the repository owner. Evidence standing in for the gate:

  • task test-check — 31848 passed, 557 skipped, 26 xfailed
  • pre-commit run --all-files — all hooks passed
  • dry-walkthrough — plan verified and stamped after three auditor passes
  • no recorded deviations from the plan

@Trecek
Trecek force-pushed the investigation-complete-marker-is-substring-matched-over-the/4386-2 branch from 019ea24 to c010e28 Compare July 29, 2026 00:41

@Trecek Trecek left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AutoSkillit PR Review — Verdict: approved_with_comments

_INLINE_CODE_RE = re.compile(r"`[^`\n]*`")


def strip_markdown_code_regions(text: str) -> str:

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[warning] bugs: strip_markdown_code_regions requires a matched opening+closing fence pair via the (|~~~).*?\1 pattern. An unclosed/odd-count fence (e.g. an issue body opens a block and never closes it) leaves everything after the dangling opening delimiter completely unstripped, including any marker text that follows it. On GitHub, an unterminated fence causes all subsequent content to render as code (until EOF), so a marker appearing there is visually in code and should not be treated as live — but detect_body_marker(body, marker) still returns True in this case, reproducing the class of false-positive this PR is meant to fix. Repro: strip_markdown_code_regions("```\ncode\n\nmore, unclosed") leaves the marker present in the stripped text. Consider treating an unmatched fence delimiter as extending to EOF (matching GitHub's rendering) so trailing content is also stripped.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Valid observation — confirmed the repro (strip_markdown_code_regions("```\ncode\n<!-- investigation_complete: true -->\nmore, unclosed") leaves the marker present). A literal fix (extend the unmatched fence to EOF, e.g. (```|~~~).*?(?:\1|\Z)) works for this repro, but strip_markdown_code_regions is a single shared implementation — tests/_helpers.py re-exports the same function for tests/docs/test_banned_phrases.py, tests/docs/test_glossary_spelling.py, tests/skills/test_investigate_design_intent_contracts.py, and tests/workspace/test_skills.py, which strip code fences from SKILL.md/docs prose before banned-phrase/mention-guard scanning. tests/test_helpers_strip.py::test_unclosed_fence_is_fail_open explicitly locks the opposite (fail-open) behavior as intentional for that use case — flipping the shared regex to fail-closed would silently drop trailing prose in doc scans, inverting the safety direction those consumers rely on. Flagging for a design decision: fork a stricter fail-closed helper scoped to detect_body_marker, or accept the shared behavior change and update the fail-open test/consumers.

Trecek and others added 4 commits July 28, 2026 21:14
Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude <noreply@anthropic.com>
@Trecek
Trecek force-pushed the investigation-complete-marker-is-substring-matched-over-the/4386-2 branch from c010e28 to d51474e Compare July 29, 2026 04:18
@Trecek
Trecek added this pull request to the merge queue Jul 29, 2026
Merged via the queue into develop with commit 8a3f184 Jul 29, 2026
3 checks passed
@Trecek
Trecek deleted the investigation-complete-marker-is-substring-matched-over-the/4386-2 branch July 29, 2026 04:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant