Purge the full visibility chain in Simulation.delete_arrays and give clones private disk views

[MaxGhenis]

Fixes the mechanism behind PolicyEngine/policyengine-us#9086: behavioral-response measurement branches recorded marginal tax rates of exactly 1.0 for every adult in both the baseline and reform branches, making relative_wage_change ≡ 0 and the substitution channel of labor supply responses (and the capital-gains MTR channel) inert in every behavioral run, while static/root-simulation MTRs stayed correct.

What was wrong

Simulation.delete_arrays called holder.delete_arrays(period) with no branch name, so only default:-keyed storage was purged. Inside a named branch (the behavioral measurement branches), the purge-then-perturb pattern used by marginal_tax_rate and friends therefore failed to remove the parent branch's cached household_net_income from the sub-branch's cloned storage; Holder.get_array's ancestor walk-up served it back, the perturbed input never fed anything, and 1 − 0/Δ = 1.0 came out for every masked adult. Simulation.derivative had the same latent bug.

Changes

1. Simulation.delete_arrays purges the caller's full visibility chain — its own branch name, each ancestor via parent_branch, and default (the existing _get_visible_branch_names, deduplicated) — in the simulation's own private storage. At root the chain is just default, so root behavior is unchanged. InMemoryStorage.delete / OnDiskStorage.delete semantics are untouched: sibling and dispatched branch keys survive (the branch-scoped-delete guarantees from the earlier C2 fix and Dispatched set_input values are dropped after cache invalidation #484 are preserved and now covered by a test).
2. Cloned holders get a private disk view. Previously Holder.clone deep-copied _memory_storage but shared the same OnDiskStorage object, so chain deletion through a clone would have mutated the parent's disk mappings. OnDiskStorage.clone() copies the _files/_enums mappings over the same storage directory, never owns directory cleanup, and keeps the original cleanup owner alive through a backreference (clone-of-clone propagates the root owner). The .npy files themselves remain shared; identical {branch}_{period} writes from two views still target the same path (pre-existing, documented in the docstring). Note memory_config defaults to None and nothing in core/-us construction enables it, so disk storage is inert in standard runs — this change is for correctness when it is enabled.

Call-site audit

All 13 simulation-level delete_arrays call sites in policyengine-us 1.729.0 (the three MTR formulas, the component-MTR helper, capital-gains responses, the SSA revenue branch calcs, and the NY CTC/EITC retained branches) are purge-for-recompute and want the new semantics; the 11 direct Holder.delete_arrays calls (pre-response input migration in system.py, BranchedSimulation cleanup) keep their existing single-branch behavior. No caller preserves a cached own/ancestor value across a purge.

Tests and checks run

• 6 new regression tests in tests/core/test_branch_scoped_delete_arrays.py: nested-branch recompute-after-purge, visibility-chain scope with sibling/dispatch survival, OnDiskStorage.clone metadata/ownership, cleanup-owner GC lifetime, child disk-delete parent-safety, and derivative() inside a named branch. All 6 fail on unfixed master and pass with this change (verified by reverting policyengine_core/ to master and rerunning).
• Full suite: 680 passed, 4 skipped, 1 xfailed; country-template runner 39 passed; make format clean.
• End-to-end against policyengine-us 1.729.0 (this branch installed editable): the #9086 repro now records measurement MTRs equal to the statics (0.1965 baseline / 0.2765 reform) and relative_wage_change −0.0996, where master records 1.0000/1.0000/0.0000.
• Towncrier fragment: changelog.d/branch-scoped-delete-arrays.fixed.md.

Observed while auditing, intentionally not addressed here

• purge_cache_of_invalid_values and policyengine-us BranchedSimulation cleanup still delete default only for named simulations (pre-existing).
• OnDiskStorage.delete removes only the exact {branch}_{period} mapping for a containing period (memory backend removes contained subperiods) and does not prune _enums.
• subsample() rebuilds the baseline branch from self.clone() (reform system) and assigns the saved baseline system to self.branches["tax_benefit_system"] — a dict key, not the branch; reproduced on the country template and will be filed separately.

🤖 Generated with Claude Code