fix: stop deriving risk from demo code, and point at the next command - #100
Merged
Conversation
Express ships examples/auth/. The ranker already deprioritizes it as example code and placed it eighth at low confidence, score 6 — but the risk map read it anyway and reported **high** authentication: authentication-related files are affected on a task about request body parsing. A high-severity risk derived from evidence the ranking itself discounted is exactly the confident-but-wrong output the diagnostics exist to prevent, and it is worse than a bad ranking because a reader has no ranked file to sanity-check it against. Risk notes now ignore demo, example, sample, and fixture directories when reading context. Changed files still count wherever they live: a diff is fact, not a guess. Verified that a real auth source file still raises the risk and a changed example still does. Also adds contextual command hints. A feature nobody discovers may as well not exist, and both --explain and verify were reachable only from --help. Saving a JSON plan now suggests the verify invocation for it, and a weak, empty, or clustered ranking suggests --explain. Hints go to stderr so reports piped to a file, parsed as JSON, or posted by the Action are unchanged, and at most one appears per run. Found by an external test report against Express. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The CLI test predated the hint and asserted stderr was empty for exactly the case that now names the follow-up command. Asserting the hint documents the behaviour instead of only permitting it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The bug: a high-severity risk from a file the ranker discounted
Express ships
examples/auth/. Asked about request body parsing, FixMap ranked that demo file eighth at low confidence, score 6, explicitly labelled "example or demo code deprioritized" — and then reported:Nothing in the task or the real answer touches authentication. This is worse than a bad ranking: the reader has no ranked file to sanity-check the claim against, and it is precisely the confident-but-wrong output the whole diagnostics layer exists to prevent.
Risk notes now ignore
demo,example,sample, andfixturedirectories when reading ranked context. Changed files still count wherever they live — a diff is fact, not a guess.Verified all three directions:
examples/auth/index.jsranked onlyexamples/auth/index.jschangedsrc/auth/login.tsrankedAlso: the CLI now points at the next useful command
Both
--explainandverifywere reachable only from--help, and a feature nobody discovers may as well not exist.Deliberate constraints: hints go to stderr, so reports piped to a file, parsed as JSON, or posted by the Action are byte-identical to before; at most one per run; and only when the situation makes the suggestion obviously useful — a strong result printed to stdout stays silent.
The README also gains a four-row command table under the install line, so the two newer commands are visible to someone scanning rather than only to someone reading.
Verification
128 core tests,
npm run cigreen, all three evaluation suites byte-identical.🤖 Generated with Claude Code