Skip to content

fix: harden peek's audit checks against false positives and blind spots - #16

Open
OnePunchMonk wants to merge 1 commit into
mainfrom
fix/peek-audit-gaps
Open

fix: harden peek's audit checks against false positives and blind spots#16
OnePunchMonk wants to merge 1 commit into
mainfrom
fix/peek-audit-gaps

Conversation

@OnePunchMonk

Copy link
Copy Markdown
Owner

Summary

  • split: fixes the embargo check being a silent no-op on datetime time_col values (it was gated on np.issubdtype(dtype, np.number), which is false for the normal datetime case — every embargo check for a typical time series never fired). Now computed in row-index space, dtype-agnostic.
  • causality: guards the flagship check against two sources of false CRITICALs — a feature_fn that changes row count (e.g. dropna()), and a non-deterministic feature_fn (unseeded randomness). Both now surface as a WARNING explaining the contract violation instead of a bogus leak finding.
  • target_leak: non-numeric feature columns were silently skipped by the correlation test; now emits a WARNING naming them, so CLEAN doesn't imply "all columns checked" when some weren't.
  • report: the CLEAN verdict now lists which checks actually ran, and adds an explicit hint when only the shallow target_leak check fired (the default when running peek audit file.csv with no feature_fn/splits/pipeline) — the case most likely to give a user false confidence.
  • cli: adds --json to peek audit, exposing the AuditReport.to_dict() output that already existed but had no CLI path (needed for CI gating).

Why

Found while reviewing peek for gaps: several of these were bugs that would silently produce wrong verdicts (embargo dead code, positional-alignment false positives) rather than missing features, which is worse for a tool whose entire value proposition is being an honest auditor.

Test plan

  • Added 7 new tests covering each fix (datetime embargo firing/not-firing, length-changing feature_fn, non-deterministic feature_fn, non-numeric column warning, CLEAN verdict naming checks run)
  • Full suite: 87 passed
  • Manually re-ran peek demo to confirm output still renders correctly end-to-end

🤖 Generated with Claude Code

- split: fix embargo check being a no-op on datetime time columns
  (previously gated on np.issubdtype(..., np.number), so it never fired
  for the normal datetime case); now computed in row-index space.
- causality: guard against false CRITICALs from feature_fn contract
  violations — length-changing feature_fn (e.g. dropna()) and
  non-deterministic feature_fn now emit a WARNING instead of a bogus leak.
- target_leak: emit a WARNING listing non-numeric columns that are
  skipped by the correlation test, instead of silently ignoring them.
- report: CLEAN verdict now names which checks actually ran, and flags
  when only the shallow target_leak check fired (the common CLI case),
  so a CLEAN result can't be mistaken for a full audit.
- cli: add --json output for CI pipeline use (AuditReport.to_dict()
  was already there but unreachable from the CLI).

Adds 7 new tests covering each fix. Full suite: 87 passed.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant