Skip to content

fix(supply-chain): surface the OSV advisory cap in the ledger and the finding - #400

Open
Mark2Mac wants to merge 1 commit into
NVIDIA:mainfrom
Mark2Mac:fix/osv-truncation-ledger
Open

fix(supply-chain): surface the OSV advisory cap in the ledger and the finding#400
Mark2Mac wants to merge 1 commit into
NVIDIA:mainfrom
Mark2Mac:fix/osv-truncation-ledger

Conversation

@Mark2Mac

Copy link
Copy Markdown
Contributor

Closes #386.

What the report says today

_fetch_vuln_details() examines at most 10 advisories per package and reports the drop through
logger.warning only. Nothing reaches the artifact:

  • analysis_completeness has no event for it, so a consumer reading execution_successful: true
    cannot tell that most of the input was never fetched;
  • the SC4 message quotes the truncated count, so a package OSV listed 153 advisories for is
    reported as 10 advisory(ies) — the number a reader uses to judge the package;
  • the finding's severity is the worst of the examined ten, so a truncated lookup can understate
    a package whose worst advisory sits past the cap.

Measured on one workstation. The larger figure in #386 (16 truncation events, 323 advisories
discarded) was taken when a bigger corpus was on disk; that cache rotates, so here is what is
reproducible today, on what survives:

truncation events in the logs 7
advisories examined 70
advisories discarded 111 (61% of 181)
worst single case 10 of 44
stored reports carrying a trace of any of it 0 of 41

That last row is the defect stated as a measurement. The mechanism is not missing: 15 of those
41 reports do carry ledger_exceptions, with size_limit, binary_content and
llm_batch_failed. Only the advisory cap has no way to appear there.

What this PR does — and does not

It does not touch the cap. A detail lookup is one HTTP request per advisory, and 153 round
trips for one package is not a trade anyone wants. The cap is a reasonable cost decision; being
invisible is the defect.

  • LedgerReason.RESULT_LIMIT, so the drop appears in analysis_completeness.ledger_exceptions
    beside the existing SIZE_LIMIT and READ_ERROR accounting. SkillSpector already accounts
    carefully for files it did not read; advisories it did not fetch belong in the same ledger.
  • observed_count / limit_count on the internal ledger event, mirroring the existing
    observed_bytes / limit_bytes. Note these stay internal: _exception_from_event keeps the
    public row narrow — what, where, why, no numbers — and real reports confirm it, size_limit
    rows carry no byte counts either. This PR does not widen that projection; changing a public
    schema is a separate decision and not one a bug fix should make quietly. The quantity
    reaches the reader through the SC4 message instead, which is where someone judging the package
    is already looking.
  • osv_client.last_truncations(), following the existing was_osv_reachable() accessor pattern,
    so the analyzer can turn the cap into evidence without changing query_batch()'s signature.
  • The advisory total is cached next to the results. A dependency listed in both
    requirements.txt and pyproject.toml is the ordinary case; without this the second lookup —
    a cache hit — would silently lose the disclosure.
  • The SC4 message quotes the number OSV reported and states how many were examined:
    153 advisory(ies) (first 10 examined): …

The severity of a truncated finding is still the worst of the examined ten. That is a
consequence of the cap; after this PR it is disclosed rather than implied.

Tests

Nine new tests, each verified red against the unmodified module before the change:

test what it pins
test_truncation_is_recorded_with_observed_and_limit the cap still holds, and it is recorded
test_no_truncation_leaves_no_record no event when nothing was dropped
test_cached_package_still_reports_its_truncation a cache hit does not lose the disclosure
test_previous_truncations_do_not_leak_into_the_next_query per-call reset
test_result_limit_event_carries_observed_and_limit_counts the two counts reach the event
test_counts_are_omitted_when_not_supplied absent stays absent, not zero
test_finding_message_states_the_true_advisory_count the message stops understating
test_truncation_surfaces_as_a_ledger_exception end-to-end through node()
test_result_limit_reaches_analysis_completeness the row survives the projection into the report

One existing test was adjusted: test_cache_hit_avoids_api_call hand-built the private cache
tuple, which now carries the advisory total. It seeds through _put_cache() instead, so it no
longer depends on a private shape it has no reason to know.

ruff check and ruff format --check clean; docker build + tests/docker/smoke.sh pass
locally (including the GitHub URL scan). The unit suite is 2216 passed / 14 skipped / 4 xfailed,
with one unrelated failure: test_mcp_stdio_initialize_registers_scan_skill times out on its
hardcoded asyncio.wait_for(..., timeout=15) when the machine is loaded. It fails on an
untouched checkout of main under the same load and passes on both when the load drops, so it
is not this branch — it may be worth a separate look, since a 15-second handshake budget will go
red on any busy CI runner.

… finding

`_fetch_vuln_details()` examines at most 10 advisories per package and reports the drop
through `logger.warning` only. Nothing reaches the artifact, so a report can state
`execution_successful: true` while most of the input was never fetched.

Three consequences, all invisible to a consumer of the JSON:

- `analysis_completeness` has no event for the drop, though the ledger already accounts
  carefully for files that were not read;
- the SC4 message quotes the truncated count, so a package OSV listed 153 advisories for
  is reported as `10 advisory(ies)` — the number a reader uses to judge the package;
- the finding's severity is the worst of the examined ten, so a truncated lookup can
  understate a package whose worst advisory sits past the cap.

Measured across 91 scan reports on one workstation: 16 truncation events, 160 advisories
examined, 323 discarded. Worst single case: 10 of 153.

The cap itself is untouched — a detail lookup is one HTTP request per advisory, and 153
round trips for one package is not a trade anyone wants. What changes is that it stops
being silent:

- `LedgerReason.RESULT_LIMIT`, with `observed_count` / `limit_count` on ledger events, so
  the drop lands in `analysis_completeness.ledger_exceptions` beside `SIZE_LIMIT`;
- `osv_client.last_truncations()`, following the existing `was_osv_reachable()` accessor
  pattern, so the analyzer can turn the cap into evidence without changing the signature
  of `query_batch()`;
- the advisory total is cached next to the results: a dependency listed in both
  requirements.txt and pyproject.toml is the ordinary case, and without this the second
  lookup — a cache hit — would silently lose the disclosure;
- the SC4 message quotes the number OSV reported and states how many were examined.

The severity of a truncated finding is still the worst of the examined ten. That is a
consequence of the cap; it is now disclosed rather than implied.

`test_cache_hit_avoids_api_call` hand-built the private cache tuple, which now carries the
advisory total. It seeds through `_put_cache()` instead, so it no longer depends on a
private shape it has no reason to know.

Closes NVIDIA#386

Signed-off-by: Marco Macrì <62335226+Mark2Mac@users.noreply.github.com>
name: str, version: str | None, examined: list[VulnResult], total: int
) -> None:
"""Note that *name* had more advisories than were examined, once per query_batch call."""
if total <= len(examined):

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

total > len(examined) does not mean the 10-result cap fired. _fetch_vuln_details() also drops individual GET failures and returns only successful parses. With 3 IDs and one 500, this records RESULT_LIMIT as “first 2 examined” even though all three were attempted and one failed; with 12 IDs plus failures it also reports the wrong cap count. Track attempted/capped IDs separately from fetch failures, use the correct failure reason, and add a partial-detail-failure regression.

@rng1995 rng1995 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[SkillSpector Review]

Requesting changes. The new truncation test conflates capped work with failed detail fetches: it infers a cap whenever successful parsed results are fewer than OSV IDs. That produces false RESULT_LIMIT metadata and incorrect examined counts on partial HTTP failures. Please track cap attempts and detail-fetch failures separately and cover the failure case described inline.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

osv_client silently drops 90% of vulnerabilities on packages with many advisories

2 participants