feat(report): opt-in deterministic version-range check for dependency findings - #963
Open
seanturner83 wants to merge 4 commits into
Open
feat(report): opt-in deterministic version-range check for dependency findings#963seanturner83 wants to merge 4 commits into
seanturner83 wants to merge 4 commits into
Conversation
… findings A dependency-CVE false positive is a factual question — is the installed version actually in the advisory's affected range? — not a code-reasoning one. Today create_dependency_report files whatever the agent passes; an agent can report a CVE against a version that's already patched past the range, or mis-attribute a CVE to the wrong package. This adds an opt-in deterministic check (no LLM) right before the dependency report is persisted, as a sibling of the existing dedup-reject: ask an advisory provider which advisories affect the exact installed version; if the cited CVE/GHSA isn't among them, reject the finding as out-of-range. - strix/report/dep_verify.py: AdvisoryProvider protocol + OsvProvider (OSV.dev /v1/query). No new dependency — uses requests (already core). - Provider-pluggable via DepVerifySettings: STRIX_DEP_VERIFY (default off), STRIX_DEP_VERIFY_PROVIDER (osv|none), STRIX_OSV_URL (point at a self-hosted OSV mirror for air-gapped / data-residency deployments — identical query contract). - Fail-open + asymmetric (never suppress a real finding): rejects ONLY on a definitive, non-empty provider answer that omits the cited advisory. Provider unreachable/error, empty result (coverage gap), missing fields, non-CVE/GHSA id, unknown ecosystem -> emit. - Off by default; when disabled the dependency path is unchanged. Tests inject a fake provider (no network) + cover provider resolution and the OSV response parse. 16 tests. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
…able in scan logs An in-range (real) finding was emitted silently, giving a scan log no evidence the version-range check ran. Add an INFO line on the confirmed-emit path, mirroring the reject / coverage-gap logs.
…rsion is clean Refine the empty-result handling: when the provider returns no advisories for the installed version, distinguish two cases via a new AdvisoryProvider.knows_package (version-less query): - provider KNOWS the package (advisories on other versions) but none affect this version -> CONFIDENT out-of-range -> reject (e.g. minimist 1.2.6 cited for CVE-2021-44906, which was fixed in 1.2.6 — OSV knows minimist, 0 affect 1.2.6). - provider doesn't know the package at all -> genuine coverage gap (private / vendored / provider lag) -> still fail-open, emit. Without this the most common dependency false positive — a real CVE cited against a version that's already patched past its range — slipped through as a 'coverage gap'. Surfaced by a live scan: a weak model, coerced into over-claiming patched minimist CVEs, had them emitted; now they're correctly rejected while the genuinely in-range lodash CVEs still emit. Also refactor OsvProvider to a shared _query helper (used by both affecting and knows_package). 18 tests. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
Contributor
Greptile SummaryThe PR adds opt-in advisory-provider verification before dependency findings are persisted.
Confidence Score: 5/5The PR appears safe to merge. No blocking failure remains; the previously reported event-loop blocking has been corrected by offloading advisory requests. Important Files Changed
Reviews (2): Last reviewed commit: "fix(dep-verify): run the advisory check ..." | Re-trigger Greptile |
verify_dependency does blocking HTTP (requests.post) and was called directly from the async create_dependency_report path — a slow/unreachable provider would block the shared agent event loop for the request timeout, stalling every concurrent agent (worst case two sequential calls on the clean-version path). - Offload the call via asyncio.to_thread so the event loop stays free. - Tighten the provider timeout 20s -> 10s (named OsvProvider._TIMEOUT_S): a single advisory lookup shouldn't hold a worker thread that long, and it fails open fast. - Add an async test asserting the verifier runs on a worker thread (not the main/ event-loop thread) and the reject still propagates. Addresses the Greptile review comment on usestrix#963. 38 report/dep tests pass. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
Collaborator
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
An opt-in, deterministic verification step for dependency-CVE findings, run
right before a
create_dependency_reportfinding is persisted — a sibling of theexisting dedup-reject at the same point. It asks an advisory provider (OSV.dev by
default) which advisories affect the exact installed version; if the finding's
cited CVE/GHSA isn't among them, the finding is out of range and is rejected.
Off by default (
STRIX_DEP_VERIFY=1to enable). No LLM. No new dependency.Why
create_dependency_reportfiles whatever the agent passes. Three false-positiveshapes slip through today:
CVE-2021-23337inlodash@4.17.21, but that version is fixed; the CVE only affects< 4.17.21.filed against
express).For any package the provider covers this is caught for free: the provider returns
the complete advisory set affecting that version, and a fabricated id simply
isn't in it → rejected. (Existence and version-membership are the same check.)
Both are factual version-range questions, not code-reasoning ones — so they're
better answered deterministically against an advisory database than by the model.
This is really anti-hallucination + anti-knowledge-cutoff grounding for
dependency claims: the model may fabricate a CVE→version match, and its training
data has a cutoff so it often doesn't reliably know which release fixed a CVE.
An advisory database is a live source of truth the model doesn't have internalised
— checking against it catches both the fabricated and the stale-knowledge cases,
while confirming the genuinely-in-range ones. This is exactly the sub-class where
an exact range check beats any LLM judgment.
"How do I know these are real and not false positives?" is a recurring user
question (e.g. #34), and today the answer is manual — review the repro steps / run
the PoC. For dependency findings that manual check is really just "is this version
in the CVE's range?", which a machine can answer exactly. This automates that one
narrow, deterministic slice — it does not touch code-reachability findings, where
manual/PoC validation still rightly applies.
How it behaves (safety first)
Fail-open and asymmetric — it never suppresses a real finding. It rejects
only on a definitive, non-empty provider answer that omits the cited advisory.
Every uncertain case emits the finding unchanged:
When disabled (the default) the dependency path is byte-for-byte unchanged.
Provider-pluggable
Strix runs in a lot of environments — air-gapped, data-residency-constrained,
orgs with their own advisory DB — so it isn't hard-wired to a hosted API:
STRIX_DEP_VERIFY_PROVIDER=osv(default) — queries an OSV-schema/v1/queryendpoint.
STRIX_OSV_URL=…— point at a self-hosted OSV mirror (identical request/response contract, so just the URL changes) for offline / residency needs.
STRIX_DEP_VERIFY_PROVIDER=none— disable.The
AdvisoryProviderprotocol makes adding another source (a private DB, anotherschema) a small, self-contained addition.
Notes
requests, already a core dep.DepVerifySettings, independent of any othertoggle.
resolution, and the OSV response parse.
Validation
vulnerable; minimist 1.2.6 = patched). In a real scan with
STRIX_DEP_VERIFY=1:the lodash CVEs were emitted (confirmed in range) and over-claimed minimist
CVEs were rejected (out of range).
out-of-range minimist CVE at all — they check the version themselves and don't
over-claim. I had to use a weaker model, coerced with an explicit
file-these-CVEs instruction, to make the reject path fire live. So in practice
the reject is a backstop for the cases a strong agent already avoids — most
valuable exactly when a cheaper/weaker model is driving the scan.
Example
Happy to adjust the config surface / naming to match your conventions, or gate it
differently — flagging early since it touches the report path.
One adjacent note: this returns a reject dict on the same path as the dedup-reject,
so it's subject to #834 (a rejected report currently rendering in the TUI as a
successfully-filed one). This PR doesn't change that rendering; when #834 is fixed
these rejects will surface correctly too. Happy to coordinate if useful.