Skip to content

feat(report): opt-in deterministic version-range check for dependency findings - #963

Open
seanturner83 wants to merge 4 commits into
usestrix:mainfrom
seanturner83:contrib/dep-version-range-verify
Open

feat(report): opt-in deterministic version-range check for dependency findings#963
seanturner83 wants to merge 4 commits into
usestrix:mainfrom
seanturner83:contrib/dep-version-range-verify

Conversation

@seanturner83

Copy link
Copy Markdown
Contributor

What

An opt-in, deterministic verification step for dependency-CVE findings, run
right before a create_dependency_report finding is persisted — a sibling of the
existing dedup-reject at the same point. It asks an advisory provider (OSV.dev by
default) which advisories affect the exact installed version; if the finding's
cited CVE/GHSA isn't among them, the finding is out of range and is rejected.

Off by default (STRIX_DEP_VERIFY=1 to enable). No LLM. No new dependency.

Why

create_dependency_report files whatever the agent passes. Three false-positive
shapes slip through today:

  • Already patched past the range — the agent reports e.g. CVE-2021-23337 in
    lodash@4.17.21, but that version is fixed; the CVE only affects < 4.17.21.
  • Mis-attributed — the agent pins a CVE to the wrong package (a lodash CVE
    filed against express).
  • Fabricated — the agent invents a real-sounding CVE id that doesn't exist.
    For any package the provider covers this is caught for free: the provider returns
    the complete advisory set affecting that version, and a fabricated id simply
    isn't in it → rejected. (Existence and version-membership are the same check.)

Both are factual version-range questions, not code-reasoning ones — so they're
better answered deterministically against an advisory database than by the model.
This is really anti-hallucination + anti-knowledge-cutoff grounding for
dependency claims: the model may fabricate a CVE→version match, and its training
data has a cutoff so it often doesn't reliably know which release fixed a CVE.
An advisory database is a live source of truth the model doesn't have internalised
— checking against it catches both the fabricated and the stale-knowledge cases,
while confirming the genuinely-in-range ones. This is exactly the sub-class where
an exact range check beats any LLM judgment.

"How do I know these are real and not false positives?" is a recurring user
question (e.g. #34), and today the answer is manual — review the repro steps / run
the PoC. For dependency findings that manual check is really just "is this version
in the CVE's range?", which a machine can answer exactly. This automates that one
narrow, deterministic slice — it does not touch code-reachability findings, where
manual/PoC validation still rightly applies.

How it behaves (safety first)

Fail-open and asymmetric — it never suppresses a real finding. It rejects
only on a definitive, non-empty provider answer that omits the cited advisory.
Every uncertain case emits the finding unchanged:

  • provider disabled / unreachable / errors / non-200 → emit
  • empty result (private/vendored package, provider coverage gap) → emit
  • missing package/version/ecosystem, or a non-CVE/GHSA identifier → emit
  • provider does list the CVE for that version → emit (confirmed real)

When disabled (the default) the dependency path is byte-for-byte unchanged.

Provider-pluggable

Strix runs in a lot of environments — air-gapped, data-residency-constrained,
orgs with their own advisory DB — so it isn't hard-wired to a hosted API:

  • STRIX_DEP_VERIFY_PROVIDER=osv (default) — queries an OSV-schema /v1/query
    endpoint.
  • STRIX_OSV_URL=… — point at a self-hosted OSV mirror (identical request/
    response contract, so just the URL changes) for offline / residency needs.
  • STRIX_DEP_VERIFY_PROVIDER=none — disable.

The AdvisoryProvider protocol makes adding another source (a private DB, another
schema) a small, self-contained addition.

Notes

  • No new dependency — the OSV provider uses requests, already a core dep.
  • New settings live in a dedicated DepVerifySettings, independent of any other
    toggle.
  • Tests inject a fake provider (no network) and cover the verdict logic, provider
    resolution, and the OSV response parse.

Validation

  • Unit tests as above (no network).
  • Observed live end-to-end on a small Node target (lodash 4.17.15 = genuinely
    vulnerable; minimist 1.2.6 = patched). In a real scan with STRIX_DEP_VERIFY=1:
    the lodash CVEs were emitted (confirmed in range) and over-claimed minimist
    CVEs were rejected (out of range).
  • Honest caveat on that run: capable models (Opus, Sonnet) declined to file the
    out-of-range minimist CVE at all
    — they check the version themselves and don't
    over-claim. I had to use a weaker model, coerced with an explicit
    file-these-CVEs instruction
    , to make the reject path fire live. So in practice
    the reject is a backstop for the cases a strong agent already avoids — most
    valuable exactly when a cheaper/weaker model is driving the scan.

Example

$ STRIX_DEP_VERIFY=1 strix --target ...
# agent reports CVE-2021-23337 in express@4.18.2  → rejected (CVE doesn't affect express)
# agent reports CVE-2021-44906 in minimist@1.2.6  → rejected (patched in 1.2.6; OSV knows minimist, 0 affect 1.2.6)
# agent reports CVE-2021-23337 in lodash@4.17.20  → emitted  (in range, real)

Happy to adjust the config surface / naming to match your conventions, or gate it
differently — flagging early since it touches the report path.

One adjacent note: this returns a reject dict on the same path as the dedup-reject,
so it's subject to #834 (a rejected report currently rendering in the TUI as a
successfully-filed one). This PR doesn't change that rendering; when #834 is fixed
these rejects will surface correctly too. Happy to coordinate if useful.

… findings

A dependency-CVE false positive is a factual question — is the installed version
actually in the advisory's affected range? — not a code-reasoning one. Today
create_dependency_report files whatever the agent passes; an agent can report a
CVE against a version that's already patched past the range, or mis-attribute a
CVE to the wrong package. This adds an opt-in deterministic check (no LLM) right
before the dependency report is persisted, as a sibling of the existing
dedup-reject: ask an advisory provider which advisories affect the exact installed
version; if the cited CVE/GHSA isn't among them, reject the finding as out-of-range.

- strix/report/dep_verify.py: AdvisoryProvider protocol + OsvProvider (OSV.dev
  /v1/query). No new dependency — uses requests (already core).
- Provider-pluggable via DepVerifySettings: STRIX_DEP_VERIFY (default off),
  STRIX_DEP_VERIFY_PROVIDER (osv|none), STRIX_OSV_URL (point at a self-hosted OSV
  mirror for air-gapped / data-residency deployments — identical query contract).
- Fail-open + asymmetric (never suppress a real finding): rejects ONLY on a
  definitive, non-empty provider answer that omits the cited advisory. Provider
  unreachable/error, empty result (coverage gap), missing fields, non-CVE/GHSA id,
  unknown ecosystem -> emit.
- Off by default; when disabled the dependency path is unchanged.

Tests inject a fake provider (no network) + cover provider resolution and the OSV
response parse. 16 tests.

🤖 Generated with [Claude Code](https://claude.com/claude-code)
…able in scan logs

An in-range (real) finding was emitted silently, giving a scan log no evidence the
version-range check ran. Add an INFO line on the confirmed-emit path, mirroring the
reject / coverage-gap logs.
…rsion is clean

Refine the empty-result handling: when the provider returns no advisories for the
installed version, distinguish two cases via a new AdvisoryProvider.knows_package
(version-less query):
- provider KNOWS the package (advisories on other versions) but none affect this
  version -> CONFIDENT out-of-range -> reject (e.g. minimist 1.2.6 cited for
  CVE-2021-44906, which was fixed in 1.2.6 — OSV knows minimist, 0 affect 1.2.6).
- provider doesn't know the package at all -> genuine coverage gap (private /
  vendored / provider lag) -> still fail-open, emit.

Without this the most common dependency false positive — a real CVE cited against
a version that's already patched past its range — slipped through as a 'coverage
gap'. Surfaced by a live scan: a weak model, coerced into over-claiming patched
minimist CVEs, had them emitted; now they're correctly rejected while the genuinely
in-range lodash CVEs still emit.

Also refactor OsvProvider to a shared _query helper (used by both affecting and
knows_package). 18 tests.

🤖 Generated with [Claude Code](https://claude.com/claude-code)
@greptile-apps

greptile-apps Bot commented Aug 3, 2026

Copy link
Copy Markdown
Contributor

Greptile Summary

The PR adds opt-in advisory-provider verification before dependency findings are persisted.

  • Adds dependency-verification settings and OSV provider resolution.
  • Verifies exact package versions and fails open when the provider cannot answer.
  • Offloads provider requests from the async agent loop and adds verdict, provider, parsing, and integration tests.

Confidence Score: 5/5

The PR appears safe to merge.

No blocking failure remains; the previously reported event-loop blocking has been corrected by offloading advisory requests.

Important Files Changed

Filename Overview
strix/config/settings.py Adds opt-in provider and endpoint configuration for dependency verification.
strix/report/dep_verify.py Implements fail-open advisory lookup, identifier normalization, and exact-version verdict logic.
strix/tools/reporting/tool.py Integrates asynchronous dependency verification immediately before report persistence.
tests/test_dep_verify.py Covers provider resolution, OSV parsing, normalization, and fail-open verdict behavior.
tests/test_reporting_fields.py Verifies that dependency advisory lookup executes outside the main event-loop thread.

Reviews (2): Last reviewed commit: "fix(dep-verify): run the advisory check ..." | Re-trigger Greptile

Comment thread strix/report/dep_verify.py Outdated
verify_dependency does blocking HTTP (requests.post) and was called directly from
the async create_dependency_report path — a slow/unreachable provider would block
the shared agent event loop for the request timeout, stalling every concurrent
agent (worst case two sequential calls on the clean-version path).

- Offload the call via asyncio.to_thread so the event loop stays free.
- Tighten the provider timeout 20s -> 10s (named OsvProvider._TIMEOUT_S): a single
  advisory lookup shouldn't hold a worker thread that long, and it fails open fast.
- Add an async test asserting the verifier runs on a worker thread (not the main/
  event-loop thread) and the reject still propagates.

Addresses the Greptile review comment on usestrix#963. 38 report/dep tests pass.

🤖 Generated with [Claude Code](https://claude.com/claude-code)
@bearsyankees

Copy link
Copy Markdown
Collaborator

@greptile

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants