Skip to content

Clear generic pileup state between records - #648

Open
SuhasSrinivasan wants to merge 1 commit into
nanoporetech:masterfrom
SuhasSrinivasan:codex/fix-generic-pileup-record-state
Open

Clear generic pileup state between records#648
SuhasSrinivasan wants to merge 1 commit into
nanoporetech:masterfrom
SuhasSrinivasan:codex/fix-generic-pileup-record-state

Conversation

@SuhasSrinivasan

@SuhasSrinivasan SuhasSrinivasan commented Aug 5, 2026

Copy link
Copy Markdown
Contributor

Fixes #647.

Summary

  • Clear all cached modification fields when the generic pileup modification iterator is exhausted.
  • Prevent a preceding read's modification call from being attributed to a following empty-MM read.
  • Add a worker-level two-record regression for the populated-to-empty state transition.

Severity

Severity: High — scientific correctness

Rationale: The affected path exits successfully but can silently inflate modified counts and reduce canonical counts in bedMethyl output.

Root cause

update_mods_iter2 populated mod_pos, canonical_base, pos_base_mod_call, and mod_strand when the iterator yielded a call, but left all four fields unchanged when it returned None. The generic worker could therefore reuse state from the preceding record.

Implementation

Clear all four cached fields on iterator exhaustion. The regression drives the actual generic worker with one explicit modified-A read followed by an implicit-mode read with an empty MM group at the same coordinate.

Preserved behavior

  • Existing behavior is unchanged while the iterator contains a call.
  • Coverage and all unrelated count categories remain unchanged.
  • No output schema or CLI option changes.

Non-goals

  • Does not redesign MM/ML parsing or generic-pileup storage.
  • Does not change optimized or high-depth pileup paths.
  • Does not include a performance refactor.

Behavior before and after

Case Before After Expected oracle
Modified read followed by empty-MM read at position 4 2 modified, 0 canonical, 100.00% 1 modified, 1 canonical, 50.00% Coverage 2 partitioned exactly 1/1
Other count categories All zero All zero Unchanged

Testing

Test environment

  • Revision tested: 948830d4e44d1be8aedadce810d1ca8068397c3a
  • Tree tested and worktree state: f0fd371c4bd0fdb6b13893c94b96fcd581748a17; clean at final verification
  • Platform: macOS on Apple silicon
  • Reference binary: installed modkit 0.6.4
  • Exact Rust and samtools versions were not retained in the original test log.
  • Dependency resolution: existing workspace manifests; no dependency or lockfile change in this PR
Test layer Exact command, fixture, or matrix Result and evidence
Core: parent-red regression Focused worker regression applied to base 5cecc3fb3a9336068d9e3c68d5c08d678153dd2c Failed as expected with n_modified = 2 instead of 1
Core: focused regression cargo test -p mod_kit generic_pileup_clears_mod_state_between_records --lib Passed
Unit/library tests cargo test -p mod_kit --lib 96 passed, 3 ignored, 0 failed
Core: affected CLI comparison Installed 0.6.4 and patched CLI on the two-record issue fixture Only the final position changed from 100.00 / 2 / 0 to 50.00 / 1 / 1
Core: applicable full workspace gate cargo test --workspace --quiet Passed; the aggregate count was not retained in the original log
Core: formatting/diff checks rustfmt --check --edition 2021 modkit-core/src/pileup/pileup_processor.rs; git diff --check Passed
Additional formatting check cargo fmt --all -- --check Reported pre-existing unrelated differences in modkit/tests/test_bedmethyl_util.rs, modkit-core/src/adjust.rs, and modkit-core/src/entropy/mod.rs; the changed file itself passed rustfmt

Tests not performed

  • No large real-data or performance/RSS benchmark was run; this change adds only constant-time field clearing on iterator exhaustion and the exact synthetic oracle covers the scientific behavior.
  • No CRAM-specific matrix was run because the defect is downstream of record decoding and format-independent.

Scientific validation

  • Count conservation: Coverage 2 equals one modified plus one canonical observation.
  • Coordinate invariant: Only reference position 4 changes.
  • Category invariant: Filtered, deletion, fail, difference, no-call, and other counts remain zero.
  • Independent oracle: Per-read MM semantics were reviewed independently; the expected fraction is exactly 1/2.

Output and compatibility

  • User-visible change is limited to correcting affected generic-pileup counts.
  • Output schema, ordering, and CLI compatibility are unchanged.
  • Unaffected records remain byte-identical in the fixture comparison.

Reviewer guide

  1. Review the two-record worker regression and its 1/1 count oracle.
  2. Review the exhausted-iterator branch in update_mods_iter2.
  3. Rerun cargo test -p mod_kit generic_pileup_clears_mod_state_between_records --lib.
  4. Confirm the one-file production change is limited to clearing per-record state.

Checklist

  • Linked issue contains reproducible observed and expected behavior.
  • Change is limited to the linked issue's scope.
  • Regression is red on the parent and green on this revision.
  • All tests performed are listed with their results.
  • Unrun applicable tests are disclosed.
  • Scientific count conservation and output compatibility are checked.
  • Changed-file formatting and diff hygiene pass.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Generic pileup can carry a modification call into the next record

1 participant