Skip to content

feat: match repeated findings across scan history - #575

Open
mldangelo-oai wants to merge 9 commits into
mdangelo/codex/scan-comparison-storagefrom
mdangelo/codex/match-repeated-findings
Open

feat: match repeated findings across scan history#575
mldangelo-oai wants to merge 9 commits into
mdangelo/codex/scan-comparison-storagefrom
mdangelo/codex/match-repeated-findings

Conversation

@mldangelo-oai

@mldangelo-oai mldangelo-oai commented Aug 20, 2026

Copy link
Copy Markdown
Collaborator

Summary

Replace the all-at-once matcher behind scans match and scans compare with a paged catalogue that reuses stable finding identities and requests full evidence only when needed.

This PR is stacked on #574. It adds a public SDK matcher but no CLI commands, flags, accepted values, environment variables, or services. It uses the existing Codex authentication.

Changes

  • Group stable finding IDs and confirmed links into compact catalogue records, then expand decisions back to the original occurrences.
  • Page catalogue records and evidence within the upstream message limit.
  • Require complete evidence before accepting a result, and reject unknown IDs, conflicting decisions, and requests that make no progress.
  • Reject empty or repeated occurrence IDs before deterministic grouping or model use.
  • Keep confirmed, uncertain, and related findings distinct while processing scan history chronologically.
  • Reject a scan-pair result when one later finding is both confirmed and uncertain, instead of caching a result that can mark another finding resolved.
  • Save complete comparisons through standard input, preserve comparisons already written when matching is canceled, and keep cancellation wired through the CLI and workbench.
  • Export matchScanFindings with self-contained public input, result, progress, and cancellation types.
  • Isolate matcher turns from repository project instructions while retaining read-only, no-approval, no-network execution.
  • Defer additional automatic matching under --max-cost to an explicit scans match --all run.

Testing

At 4110ecec1822c5c09f516e7fa4b61ca6eb8ae185:

  • Focused comparison, catalogue, property, API, public-surface, and sealed-history tests: 93 passed, 3 Windows skips.
  • Full SDK suite with Node 22.13.0, Bun 1.3.14, and seed 12345: 1,673 passed, 28 platform or integration skips, 0 failed.
  • Type checks, Prettier, git diff --check, and the SDK build passed.
  • Package inspection and installed SDK, CLI, NodeNext type, bundled Codex, and nested-worker smoke tests passed: 259 package entries and 111 plugin files.
  • Regression coverage includes invalid occurrence identities, conflicting uncertainty, omitted evidence, complete scan inputs, forced recomputation, cancellation, invalid responses, package type declarations, and unchanged scan artifacts.
  • Fresh exact-head GitHub CI and Codex review are pending after this update.

Risk and rollout

Merge #574 first, then retarget this PR to main.

A wrong confirmed match can affect later comparisons. Existing saved comparisons are not rewritten on upgrade; use scans match --all --force to revisit an earlier result. Paging respects the upstream message limit but does not establish matching accuracy or provide unlimited context.

With --max-cost or maxCostUsd, matching that needs more context is deferred instead of saving a partial result or making extra automatic calls.

Public disclosure review

Historical public corporate Git authorship and generic Codex product-help links were reviewed; neither contains nonpublic material. This update adds no sensitive material.

  • No customer, partner, prospect, or user identities, data, or identifying details are included.
  • No credentials, personal data, private source, scan findings, or nonpublic links or tickets are included.
  • I reviewed the branch name, title, description, commits, changes, comments, logs, screenshots, attachments, and links for public disclosure.

@mldangelo-oai

Copy link
Copy Markdown
Collaborator Author

@codex review

Please review the current head, 5875cb27239e4a310be1bc3cd4c58b67275b5319, against #574. This is the matcher and existing CLI/SDK integration split from #567. The storage contract is reviewed separately in #574.

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. 🚀

Reviewed commit: 5875cb2723

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@mldangelo-oai

Copy link
Copy Markdown
Collaborator Author

@codex review

The description now includes the final QA results and merge order. Please review the current head, 5875cb27239e4a310be1bc3cd4c58b67275b5319. The code has not changed since the earlier review.

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. Keep them coming!

Reviewed commit: 5875cb2723

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Aug 20, 2026

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review Completed 2026-08-22T15:04:24.156822Z 4110ece Manual request
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: c3d4a5661b

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread sdk/typescript/src/scan-comparison.ts

Copy link
Copy Markdown
Collaborator Author

@codex review

Please review the current head, c3d4a5661b31870dd2960432f3390f491af5d9ee. The PR description includes the latest cleanup and QA results.

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. Can't wait for the next one!

Reviewed commit: c3d4a5661b

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Copy link
Copy Markdown
Collaborator Author

@codex review

Please review the current head, c8b75fc305061c6fa0a39e842f111a0d9c9707e4. This also fixes the omitted-evidence case raised in the previous review and adds tests for all four final outcomes.

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. You're on a roll.

Reviewed commit: c8b75fc305

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@mldangelo-oai

Copy link
Copy Markdown
Collaborator Author

@codex review

Please review the current head, 1b1e546213d1a041e3daf03985f24d24b8ade8a8, against #574. This additive update merges current main, preserves failed-history output semantics, recomputes forced single-pair matches without old semantic groups, and compares complete selected scans before caching automatic results. The PR description includes the regressions, package checks, synthetic live QA, and the remaining full-suite checks.

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. 🎉

Reviewed commit: 1b1e546213

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@mldangelo-oai

Copy link
Copy Markdown
Collaborator Author

@codex review

The description now includes the completed full-suite runs, installed-package and synthetic live QA, and the green exact-head CI result. Please review the current head, 1b1e546213d1a041e3daf03985f24d24b8ade8a8, against #574. The code has not changed since the previous review.

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. Already looking forward to the next diff.

Reviewed commit: 1b1e546213

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread sdk/typescript/src/scan-comparison.ts Outdated
@mldangelo-oai mldangelo-oai changed the title feat: match repeated findings across scans feat: match repeated findings across scan history Aug 22, 2026
@mldangelo-oai

Copy link
Copy Markdown
Collaborator Author

@codex review

Please review exact head 043fa2a80cbfe13cc9ae640339ce92dcb60e9d63 against #574. This update resolves the stacked merge, rejects confirmed and uncertain conflicts before caching, preserves cross-scan uncertainty, uses standard input for saved comparisons, and keeps the published matcher types self-contained.

@mldangelo-oai

Copy link
Copy Markdown
Collaborator Author

@codex security review

Please run a security review on exact head 043fa2a80cbfe13cc9ae640339ce92dcb60e9d63 against #574. Focus on scan-history persistence, model-result validation, cancellation, standard-input transport, and the public SDK boundary.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 043fa2a80c

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

if (input.before.length === 0 || input.after.length === 0) {
return { matches: [], uncertain: [] };
}
const known = reconcileComparison(

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Reject occurrence-ID collisions before reconciliation

The newly exported SDK accepts imported finding arrays without verifying that occurrenceId values are unique across both sides. If malformed before/after documents reuse an ID for unrelated findings, groupFindings treats them as the same occurrence node, so this initial reconciliation can set known.complete and return a high-confidence match without consulting the model. Validate nonempty, globally unique occurrence IDs before reconciling to prevent unrelated scan history from being merged.

AGENTS.md reference: sdk/typescript/AGENTS.md:L22-L24

Useful? React with 👍 / 👎.

export type { ScanCost, ScanSessionEvent } from "./cost.js";
export type { CustomValidationResult } from "./custom-validation.js";
export type { ScanActivity, ScanActivityStatus } from "./scan-activity.js";
export { matchScanFindings } from "./scan-comparison.js";

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Disable repository instructions for the public matcher

When an SDK caller follows the new documented flow and sets workingDirectory to the scanned repository, Codex automatically loads applicable repository AGENTS.md files as developer instructions. Exporting matchScanFindings in this state lets repository-controlled instructions outrank the prompt's untrusted-data warning and steer high-confidence matches; callers that persist the result can then corrupt scan history. Disable project-document loading for comparison threads or use an instruction-free working directory before exposing this matcher.

AGENTS.md reference: sdk/typescript/AGENTS.md:L8-L12

Useful? React with 👍 / 👎.

…/pr-575

# Conflicts:
#	sdk/typescript/scripts/fixtures/package-consumer.ts
@mldangelo-oai

Copy link
Copy Markdown
Collaborator Author

@codex review

Please review exact head 4110ecec1822c5c09f516e7fa4b61ca6eb8ae185 against #574.

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. Keep it up!

Reviewed commit: 4110ecec18

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants