Skip to content

fix: stop deriving risk from demo code, and point at the next command - #100

Merged
aryamthecodebreaker merged 2 commits into
mainfrom
fix/risk-from-demo-code
Jul 26, 2026
Merged

fix: stop deriving risk from demo code, and point at the next command#100
aryamthecodebreaker merged 2 commits into
mainfrom
fix/risk-from-demo-code

Conversation

@aryamthecodebreaker

Copy link
Copy Markdown
Owner

The bug: a high-severity risk from a file the ranker discounted

Express ships examples/auth/. Asked about request body parsing, FixMap ranked that demo file eighth at low confidence, score 6, explicitly labelled "example or demo code deprioritized" — and then reported:

## Risk Map
- **high** authentication: authentication-related files are affected

Nothing in the task or the real answer touches authentication. This is worse than a bad ranking: the reader has no ranked file to sanity-check the claim against, and it is precisely the confident-but-wrong output the whole diagnostics layer exists to prevent.

Risk notes now ignore demo, example, sample, and fixture directories when reading ranked context. Changed files still count wherever they live — a diff is fact, not a guess.

Verified all three directions:

Input Risk
examples/auth/index.js ranked only none ✅
examples/auth/index.js changed authentication ✅
src/auth/login.ts ranked authentication ✅

Also: the CLI now points at the next useful command

Both --explain and verify were reachable only from --help, and a feature nobody discovers may as well not exist.

$ fixmap plan --issue "..." --format json --output plan.json

After you make the change, check it against this plan:
  fixmap verify --report plan.json --diff main...HEAD
$ fixmap plan --issue "improve DX when running into errors"

Expected a file that is not listed? Ask why it was left out:
  fixmap plan --issue "<same task>" --explain <path>

Deliberate constraints: hints go to stderr, so reports piped to a file, parsed as JSON, or posted by the Action are byte-identical to before; at most one per run; and only when the situation makes the suggestion obviously useful — a strong result printed to stdout stays silent.

The README also gains a four-row command table under the install line, so the two newer commands are visible to someone scanning rather than only to someone reading.

Verification

128 core tests, npm run ci green, all three evaluation suites byte-identical.

🤖 Generated with Claude Code

aryamthecodebreaker and others added 2 commits July 26, 2026 20:07
Express ships examples/auth/. The ranker already deprioritizes it as example
code and placed it eighth at low confidence, score 6 — but the risk map read
it anyway and reported

  **high** authentication: authentication-related files are affected

on a task about request body parsing. A high-severity risk derived from
evidence the ranking itself discounted is exactly the confident-but-wrong
output the diagnostics exist to prevent, and it is worse than a bad ranking
because a reader has no ranked file to sanity-check it against.

Risk notes now ignore demo, example, sample, and fixture directories when
reading context. Changed files still count wherever they live: a diff is
fact, not a guess. Verified that a real auth source file still raises the
risk and a changed example still does.

Also adds contextual command hints. A feature nobody discovers may as well
not exist, and both --explain and verify were reachable only from --help.
Saving a JSON plan now suggests the verify invocation for it, and a weak,
empty, or clustered ranking suggests --explain. Hints go to stderr so
reports piped to a file, parsed as JSON, or posted by the Action are
unchanged, and at most one appears per run.

Found by an external test report against Express.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The CLI test predated the hint and asserted stderr was empty for exactly the
case that now names the follow-up command. Asserting the hint documents the
behaviour instead of only permitting it.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@aryamthecodebreaker
aryamthecodebreaker merged commit eda6b0a into main Jul 26, 2026
2 checks passed
@aryamthecodebreaker
aryamthecodebreaker deleted the fix/risk-from-demo-code branch July 26, 2026 14:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant