fix(supply-chain): surface the OSV advisory cap in the ledger and the finding - #400
fix(supply-chain): surface the OSV advisory cap in the ledger and the finding#400Mark2Mac wants to merge 1 commit into
Conversation
… finding `_fetch_vuln_details()` examines at most 10 advisories per package and reports the drop through `logger.warning` only. Nothing reaches the artifact, so a report can state `execution_successful: true` while most of the input was never fetched. Three consequences, all invisible to a consumer of the JSON: - `analysis_completeness` has no event for the drop, though the ledger already accounts carefully for files that were not read; - the SC4 message quotes the truncated count, so a package OSV listed 153 advisories for is reported as `10 advisory(ies)` — the number a reader uses to judge the package; - the finding's severity is the worst of the examined ten, so a truncated lookup can understate a package whose worst advisory sits past the cap. Measured across 91 scan reports on one workstation: 16 truncation events, 160 advisories examined, 323 discarded. Worst single case: 10 of 153. The cap itself is untouched — a detail lookup is one HTTP request per advisory, and 153 round trips for one package is not a trade anyone wants. What changes is that it stops being silent: - `LedgerReason.RESULT_LIMIT`, with `observed_count` / `limit_count` on ledger events, so the drop lands in `analysis_completeness.ledger_exceptions` beside `SIZE_LIMIT`; - `osv_client.last_truncations()`, following the existing `was_osv_reachable()` accessor pattern, so the analyzer can turn the cap into evidence without changing the signature of `query_batch()`; - the advisory total is cached next to the results: a dependency listed in both requirements.txt and pyproject.toml is the ordinary case, and without this the second lookup — a cache hit — would silently lose the disclosure; - the SC4 message quotes the number OSV reported and states how many were examined. The severity of a truncated finding is still the worst of the examined ten. That is a consequence of the cap; it is now disclosed rather than implied. `test_cache_hit_avoids_api_call` hand-built the private cache tuple, which now carries the advisory total. It seeds through `_put_cache()` instead, so it no longer depends on a private shape it has no reason to know. Closes NVIDIA#386 Signed-off-by: Marco Macrì <62335226+Mark2Mac@users.noreply.github.com>
| name: str, version: str | None, examined: list[VulnResult], total: int | ||
| ) -> None: | ||
| """Note that *name* had more advisories than were examined, once per query_batch call.""" | ||
| if total <= len(examined): |
There was a problem hiding this comment.
total > len(examined) does not mean the 10-result cap fired. _fetch_vuln_details() also drops individual GET failures and returns only successful parses. With 3 IDs and one 500, this records RESULT_LIMIT as “first 2 examined” even though all three were attempted and one failed; with 12 IDs plus failures it also reports the wrong cap count. Track attempted/capped IDs separately from fetch failures, use the correct failure reason, and add a partial-detail-failure regression.
rng1995
left a comment
There was a problem hiding this comment.
[SkillSpector Review]
Requesting changes. The new truncation test conflates capped work with failed detail fetches: it infers a cap whenever successful parsed results are fewer than OSV IDs. That produces false RESULT_LIMIT metadata and incorrect examined counts on partial HTTP failures. Please track cap attempts and detail-fetch failures separately and cover the failure case described inline.
Closes #386.
What the report says today
_fetch_vuln_details()examines at most 10 advisories per package and reports the drop throughlogger.warningonly. Nothing reaches the artifact:analysis_completenesshas no event for it, so a consumer readingexecution_successful: truecannot tell that most of the input was never fetched;
reported as
10 advisory(ies)— the number a reader uses to judge the package;a package whose worst advisory sits past the cap.
Measured on one workstation. The larger figure in #386 (16 truncation events, 323 advisories
discarded) was taken when a bigger corpus was on disk; that cache rotates, so here is what is
reproducible today, on what survives:
That last row is the defect stated as a measurement. The mechanism is not missing: 15 of those
41 reports do carry
ledger_exceptions, withsize_limit,binary_contentandllm_batch_failed. Only the advisory cap has no way to appear there.What this PR does — and does not
It does not touch the cap. A detail lookup is one HTTP request per advisory, and 153 round
trips for one package is not a trade anyone wants. The cap is a reasonable cost decision; being
invisible is the defect.
LedgerReason.RESULT_LIMIT, so the drop appears inanalysis_completeness.ledger_exceptionsbeside the existing
SIZE_LIMITandREAD_ERRORaccounting. SkillSpector already accountscarefully for files it did not read; advisories it did not fetch belong in the same ledger.
observed_count/limit_counton the internal ledger event, mirroring the existingobserved_bytes/limit_bytes. Note these stay internal:_exception_from_eventkeeps thepublic row narrow — what, where, why, no numbers — and real reports confirm it,
size_limitrows carry no byte counts either. This PR does not widen that projection; changing a public
schema is a separate decision and not one a bug fix should make quietly. The quantity
reaches the reader through the SC4 message instead, which is where someone judging the package
is already looking.
osv_client.last_truncations(), following the existingwas_osv_reachable()accessor pattern,so the analyzer can turn the cap into evidence without changing
query_batch()'s signature.requirements.txtandpyproject.tomlis the ordinary case; without this the second lookup —a cache hit — would silently lose the disclosure.
153 advisory(ies) (first 10 examined): …The severity of a truncated finding is still the worst of the examined ten. That is a
consequence of the cap; after this PR it is disclosed rather than implied.
Tests
Nine new tests, each verified red against the unmodified module before the change:
test_truncation_is_recorded_with_observed_and_limittest_no_truncation_leaves_no_recordtest_cached_package_still_reports_its_truncationtest_previous_truncations_do_not_leak_into_the_next_querytest_result_limit_event_carries_observed_and_limit_countstest_counts_are_omitted_when_not_suppliedtest_finding_message_states_the_true_advisory_counttest_truncation_surfaces_as_a_ledger_exceptionnode()test_result_limit_reaches_analysis_completenessOne existing test was adjusted:
test_cache_hit_avoids_api_callhand-built the private cachetuple, which now carries the advisory total. It seeds through
_put_cache()instead, so it nolonger depends on a private shape it has no reason to know.
ruff checkandruff format --checkclean;docker build+tests/docker/smoke.shpasslocally (including the GitHub URL scan). The unit suite is 2216 passed / 14 skipped / 4 xfailed,
with one unrelated failure:
test_mcp_stdio_initialize_registers_scan_skilltimes out on itshardcoded
asyncio.wait_for(..., timeout=15)when the machine is loaded. It fails on anuntouched checkout of
mainunder the same load and passes on both when the load drops, so itis not this branch — it may be worth a separate look, since a 15-second handshake budget will go
red on any busy CI runner.