feat(3.9): tail-df consolidation - three-value df_convention knob + t(residual) defect fix (M-127)

CHANGELOG.md

@@ -8,6 +8,25 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 ## [Unreleased]
 
 ### Fixed
+- **Normal-theory tail inference on clustered analytical paths — the D4 +
+  tail-df defect fix of the 3.9 variance-consolidation program** ([M-127]).
+  Four surfaces previously passed `df=None` (silent normal-theory z) to
+  their analytical p-value/CI computation on clustered fits: `WooldridgeDiD`
+  default-`hc1` (within and `hc1`+`cohort_trends` arms), `StackedDiD`,
+  `SunAbraham` aggregates — the D4 defect: one fit reported residual-t
+  per-cohort cells but z aggregates — and the `ImputationDiD` pretrends lead
+  regression. At |t| = 2, z understates the t(G−1) p-value by 24.2% at
+  G=20 and 13.3% at G=40. All four now use **t(residual df)** under the new
+  `"residual"` default (aggregates share the cells' df source on
+  SunAbraham; classical/hc2 arms and every SE/point estimate are
+  bit-identical; the movement is z → t at the fitted residual df — the G−1
+  convergence with fixest/Stata is the v4 default flip, M-004..M-006 +
+  M-128..M-131). `df_convention="normal"` reproduces the pre-3.9 numbers on
+  the affected lanes. Wild-cluster-bootstrap p/CIs (test-inversion) and all
+  influence-function estimators are unaffected. `WooldridgeDiD` post-fit
+  `aggregate()` reproduces fit-time inference on every OLS arm (the
+  internal `_df_one_way` generalized to `_df_analytic_fallback`, with a
+  `__setstate__` pickle migration).
 - **Clustered CR1 SEs converge on `K_reference` — the D1+D2 fix of the 3.9
   variance-consolidation program** ([M-126]). The clustered `hc1` CR1
   finite-sample factor `(G/(G−1))·((n−1)/(n−k))` now uses
@@ -40,7 +59,8 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
   `ssc(K.fixef="full", K.exact=TRUE)` at ~1e-16, with the one-df deviation of
   the nested composition from both references (neither implements
   nested-drop + exact-remainder) pinned exactly in two new goldens. Tail df
-  conventions are deliberately unchanged (PR C of the program). The
+  conventions were deliberately unchanged by this fix and converged
+  separately (the [M-127] entry above). The
   `docs/benchmarks.rst` TWFE "SE Rel Diff 0.1%" cell is generated by the
   R-dependent benchmark refresh and still shows the pre-fix value; the
   refresh re-run is tracked in TODO.md (expected movement 0.1% → 0.0%).
@@ -65,8 +85,9 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
   convention passed to inference, with each cell classified as a documented
   defect or a declared legitimate exception with its reason. The clustered
   CR1 `k` cells now pin the converged `K_reference` accounting (see the
-  [M-126] entry above); the remaining defect rows are the tail-df items
-  scheduled for PR C of the 3.9 variance-consolidation program.
+  [M-126] entry above), and the tail-df cells pin the converged
+  `df_convention` resolution (the [M-127] entry above) — the matrix carries
+  no remaining defect rows.
 - **`WooldridgeDiD` silently dropped genuine post-treatment effects
   (issue #724).** With `control_group="never_treated"`, the ETWFE design emitted
   every cohort×time indicator *including* the reference period — which is
@@ -194,6 +215,19 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
   rejecting the design.
 
 ### Added
+- **`df_convention` becomes a library-wide three-value knob** ([M-127]).
+  `{"residual", "cluster", "normal"}` is now accepted everywhere the knob
+  exists, and the knob extends to five standalone estimators: `SunAbraham`,
+  `WooldridgeDiD` (OLS arms), `StackedDiD`, `ImputationDiD` (pretrends lead
+  regression), and `LPDiD` (default `"cluster"` — its Stata-anchored t(G−1)
+  reference, unchanged). `"cluster"` = the Stata/fixest `G − 1` reference
+  (the v4 default); `"normal"` = deliberate normal-theory z at the fallback
+  level on every fit (also the exact-reproducibility path for pre-3.9
+  numbers on the surfaces fixed by the [M-127] entry under Fixed). Survey/
+  replicate df and Bell-McCaffrey DOF keep precedence under every value;
+  point estimates, SEs, and t-statistics never move. Results objects echo
+  `df_convention`, and `SunAbraham`/`StackedDiD` gain `inference_df` (the
+  overall-ATT df actually used; None under bootstrap overrides).
 - **Stata parity arm for ETWFE and Callaway-Sant'Anna ATT(g,t) (`jwdid` / `csdid`).**
   `benchmarks/stata/generate_etwfe_cs_golden.do` anchors both staggered
   estimators against their canonical Stata implementations on the genuine

DEFERRED.md

@@ -113,7 +113,7 @@ here carry the remaining implementation work and cross-link their `M-xxx` ids.
 
 | Issue | Location | PR | Priority |
 |-------|----------|----|----------|
-| Clustered-CR1 inference df **default flip to `"cluster"` (G−1) at v4** — the opt-in `df_convention=` knob landed 2026-07 (DiD/TWFE/MPD + LinearRegression; REGISTRY §TwoWayFixedEffects deviation note); the remaining work is the major-version default change (moves every clustered p-value/CI) + migration note + flipping `TestDfConvention`/`test_moderate_t_pins_residual_df_convention` expectations. Also evaluate extending the knob to standalone estimators with CR1-t inference at that time. Lifecycle tracked in docs/v4-deprecations.yaml (M-004..M-006). | `diff_diff/linalg.py::LinearRegression`, `diff_diff/estimators.py`, `diff_diff/twfe.py` | — | Medium |
+| Clustered inference df **default flip to `"cluster"` (G−1) at v4** — the `df_convention=` knob is now the library-wide THREE-VALUE surface (PR C / M-127, 3.9: DiD/TWFE/MPD/LinearRegression + SunAbraham/WooldridgeDiD-OLS/StackedDiD/ImputationDiD-pretrends at default `"residual"`, LPDiD already at `"cluster"`); the remaining work is the major-version default change (moves every clustered p-value/CI) + migration note + flipping `TestDfConvention`/`test_moderate_t_pins_residual_df_convention`/the per-estimator knob suites' expectations. Lifecycle tracked in docs/v4-deprecations.yaml (M-004..M-006 + M-128..M-131). | `diff_diff/linalg.py::LinearRegression`, `diff_diff/estimators.py`, `diff_diff/twfe.py`, `diff_diff/sun_abraham.py`, `diff_diff/wooldridge.py`, `diff_diff/stacked_did.py`, `diff_diff/imputation.py` | — | Medium |
 
 ## Decision record — won't-fix / waived
 

TODO.md

@@ -34,7 +34,7 @@ Related tracking surfaces:
 | `WooldridgeDiD` REFUSES a panel whose units split into disconnected support groups within a cohort, rather than estimating what IS identified. The connectivity guard (REGISTRY *within-cohort support connectivity*) correctly detects that a closed component's cells are collinear with the unit FE — previously QR dropped one silently and the overall ATT averaged an incomplete set (issue #724's failure mode via unit support). **Refusing is the safe answer, not the complete one.** The connected component containing the reference is still fully identified, so the estimable resolution is either (a) estimate the connected component and report the disconnected units as excluded, with the estimand restated (a sub-population of units, so it needs a REGISTRY definition and interacts with the survey-domain row above), or (b) per-component references, if a component with its own pre-period can carry its own normalization — needs a methodology decision, since components then are not comparable on one baseline. Gate with the split-support fixture in `TestWithinCohortSupportConnectivity`. | `diff_diff/wooldridge.py` | #724-codex-R7 | Heavy | Medium |
 | `WooldridgeDiD` fully resolves the `SurveyDesign` TWICE on every supported survey fit. The pre-exclusion validation pass (added so invalid metadata cannot hide in rows that cohort exclusion deletes) calls `survey_design.resolve(sample)`, and each fitter then calls `_resolve_survey_for_wooldridge` -> `_resolve_survey_for_fit` on the same frame, repeating weight normalization, strata/PSU/FPC validation and design-array construction. Any fit that REACHES the second resolve has an unchanged sample (survey + unidentified-cohort exclusion raises first), so the first result is reusable: capture the `_resolve_survey_for_fit` 4-tuple early and thread it into the three fitters as an optional `pre_resolved`. **Caveat that makes this non-trivial:** `sample = sample.reset_index(drop=True)` runs BETWEEN the two calls, so the reused object must be verified index-independent (resolution extracts positional numpy arrays, but `_inject_cluster_as_psu` and the metadata recompute need checking), and the early call must stop suppressing warnings or the user loses the weight-normalization notice. Gate with a survey fit asserting one normalization warning and byte-identical SEs. | `diff_diff/wooldridge.py` | #724-codex-R10 | Mid | Low |
 | Bootstrap re-aggregation for `CallawaySantAnnaResults.aggregate()` — a bootstrapped fit currently RAISES rather than substituting analytical inference for percentile-bootstrap statistics. The value-bound `BootstrapReplaySpec` (bit-identical replay, picklable, immune to post-fit `set_params`) is already in-tree and spike-verified; wiring it needs the per-`(g,t)` / per-event-time draw retention plus `assert_allclose` parity tests against the fit-time bootstrap numbers (NOT bit-identity — the fused GEMM's column count differs post-fit, ~1 ULP reassociation). | `diff_diff/aggregation.py`, `diff_diff/staggered_results.py` | #726 | Mid | Medium |
-| Consolidate the inference-df precedence duplicated across `honest_did.py` (3 copies at ~L655/L836/L1004) onto the shared `resolve_inference_df()` helper added in `diff_diff/aggregation.py`. The copies are correct today; the risk is drift if the survey/replicate precedence changes in one place only. | `diff_diff/honest_did.py` | #726 | Quick | Low |
+| Consolidate the inference-df precedence duplicated across `honest_did.py` (3 copies at ~L655/L836/L1004) onto the shared `resolve_inference_df()` helper added in `diff_diff/aggregation.py`. The copies are correct today; the risk is drift if the survey/replicate precedence changes in one place only. (Adjacent but distinct from PR C's `utils.resolve_tail_df`: that is the FIT-TIME `df_convention` fallback resolver, this is a post-fit results READER.) | `diff_diff/honest_did.py` | #726 | Quick | Low |
 | `ContinuousDiD` CGBS-2024 remaining extensions (earlier phases — `covariates=` reg/dr, `treatment_type="discrete"`, single-cohort `control_group="lowest_dose"` with estimand `ATT(d)−ATT(d_L)` — are already supported; see REGISTRY Note #7). Remaining (all deferred `NotImplementedError`, documented): `estimation_method="ipw"` on the dose curve (scalar-adjustment / degenerate); `covariates=` × `survey_design=` (weighted OR + weighted nuisance IF); multi-cohort **heterogeneous-support** discrete aggregation (support-aware: average each dose only over the cohorts that observe it); **multi-cohort `lowest_dose`** (within-cohort `d_L` reference + support-aware cross-cohort aggregation); and **`covariates=` × `lowest_dose`** (conditional-PT-relative-to-`d_L` estimand). Single-cohort / 2-period / shared-support multi-cohort are supported. | `continuous_did.py` | CGBS-2024 | Heavy | Low |
 | `WooldridgeDiD` does not apply the W2025 Sec 5.4 `D_{G_max} x X` covariate normalization, and three sibling covariate rank deficiencies are pre-existing. Measured with the period range pinned and only the never-treated units toggled: (1) time-invariant `exovar` is absorbed by the unit FE, 4 of 26 columns, IDENTICALLY with and without never-treated units; (2) `xgvar`'s cell x covariate block, 19 of 41, identical on both panels; (3) `xtvar` under `demean_covariates=False` does exhibit the `sum_g D_g x = x` dependency that the default demeaning removes; (4) the newly-reachable case -- time-VARYING data passed through `exovar`, which its own docstring reserves for time-invariant covariates -- where the paper's `dT_i` rule would give a deterministic `D_{G_max} x X` drop instead of QR's arbitrary pick (coefficients unaffected, `1.35e-14`; `rank_deficient_action="error"` raises). REGISTRY's narrowed Sec 5.4 note cross-references this row. **Trap for whoever takes it:** `xtvar` under the DEFAULT `demean_covariates=True` is FULL RANK -- the raw block carries demeaned values while `D_g x X` carries raw ones -- and forcing the drop there moves `overall_att` 1.11903 -> 1.46269. Pinned as-is by `TestComparisonSupportFiltering::test_cells_derived_groups_did_not_leak_into_the_design`. | `diff_diff/wooldridge.py` | #729-followup | Heavy | Medium |
 | `WooldridgeDiD.n_control_units` counts never-treated UNITS on `control_group="never_treated"` regardless of method, but on the nonlinear paths (`logit`/`poisson`) treated units' pre-treatment rows ARE the identifying comparison -- only the OLS path absorbs them into their own cells. So the reported count under-states the comparison pool exactly where the REGISTRY control-pool asymmetry note applies. Widen to `not_yet_treated or (never_treated and method != "ols")`, or document the count as never-treated-units-by-definition. Behavior is PRE-EXISTING; documented for now in the REGISTRY control-pool Note rather than changed, because widening moves a public results field and wants its own ledger row and test matrix. | `diff_diff/wooldridge.py` | #729-followup | Mid | Low |
@@ -64,6 +64,7 @@ generic sparse-FE, QR+SVD rank-detection redundancy, `check_finite` bypass — m
 | Section-8 naming-completeness guard test: sweep the public surface (module-level functions, class `__init__`/`fit` params, results dataclass fields) for contract-rename violations and fail unless each hit is either a `docs/v4-deprecations.yaml` row or an allowlist entry carrying a stated reason (seed the allowlist with the documented domain-vocabulary exemptions: the staggered family's ATT(g,t) `group`/`groups`, `TripleDifference.fit[group]`, wrapper params dying with `M-070..M-077`, and params inherited from `DifferenceInDifferences.__init__`). Today the rules are normative prose and the ledger holds only what an audit remembered — two successive manual sweeps each found surfaces the prior one missed (`M-094`/`M-095`, then the 19-row `M-097..M-115` function sweep). Should land BEFORE Phase 2c so the rename PR works from a mechanically-verified list. Two further checks belong in the same guard, each having been missed by hand across successive review rounds: (a) **phase-table agreement** — every row's `phase` must appear in the matching `docs/v4-design.md` section 9 table entry and vice versa (the section 9 checklist asserts this agreement but nothing verifies it, and three separate rounds caught a stale table); (b) **consumer coverage** (section 8 rule 11) — for each rename row, grep `diff_diff/` and `docs/methodology/` for the old name and require every hit to be either in the row's `code_refs` or allowlisted, since `getattr(obj, "old_name", default)` degrades silently rather than raising after removal. | `tests/`, `docs/v4-design.md` | gating-completeness amendment | Mid | Medium |
 | Tracking-file contract guard test: reject NEW active deferred-work pointers at `TODO.md` (deferred rows live in `DEFERRED.md`; allowlist for historical/past-tense prose and actionable-row pointers) and assert rows cross-linking a `docs/v4-deprecations.yaml` `M-xxx` id don't restate ledger status. Origin: tracking-split local review R2. | `tests/`, `TODO.md`, `DEFERRED.md` | tracking-split | Quick | Low |
 | Real-data CI canary for dataset-backed replication tests: `test_methodology_lwdid.py`'s Prop 99 / Walmart goldens skip (visibly) when loaders fall back to synthetic; add a lane or canary asserting `df.attrs["source"] == "lwdid_ssc_ancillary"` in CI so network regressions cannot silently de-gate the replication tests. Follow-on from the loader-fallback repair (#723), which made provenance explicit but deliberately did not add a network-dependent CI lane. | `tests/test_methodology_lwdid.py`, `.github/workflows/` | LWDiD validation suite | Quick | Low |
+| df-provenance completion follow-up (PR C descoping): a Wooldridge public per-row df channel (a per-key dict that can also carry the hc2_bm BM contrast dofs the scalar `_df_analytic_fallback` cannot represent — the unified surface's Wooldridge df column stays NaN until then); ImputationDiD lead-horizon `event_study_df` (needs the `_compute_lead_coefficients`/`_aggregate_event_study` return-path change plus bootstrap-clearing semantics for post rows); and the `inference_df` (DiD/MPD/SA/StackedDiD) vs `df_inference` (CS/dCDH) adapter-naming unification. | `diff_diff/wooldridge_results.py`, `diff_diff/imputation.py`, `diff_diff/results_base.py` | #variance-PR-C | Mid | Low |
 | `worktree-rm` safety via a tested argv helper: the prose rewrite (ask-before-remove confirmation gate, detached-HEAD reachability/rescue, tip-identity force-delete guard) was reverted to the main version because editing the prose repeatedly reintroduced shell-injection (last: sourcing a state file built from a git-derived branch name). Restore those guards in a `worktree_rm.py` that takes the name via file ingress, invokes git through argv arrays, and has a metacharacter-branch/path injection regression test — the pattern that worked for `pr_prepare.py`/`premerge_scan.py`. | `.claude/commands/worktree-rm.md`, `.claude/scripts/` | skill-audit | Heavy | Medium |
 | `premerge_scan.py` should scan the staged blob (`git show :path`) for staged methodology files, not the working-tree copy — a stage-then-revert-working-copy edit currently reads the safe working version and misses the staged violation. Union staged-index findings with unstaged/untracked filesystem findings. | `.claude/scripts/premerge_scan.py` | skill-audit | Mid | Low |
 | Re-add committed-range methodology scanning to `/push-pr-update` §3b (clean tree, commits ahead) using `premerge_scan.py --range`, with the comparison ref passed as **data** (resolved into a quoted variable in one Bash call, never a raw `<placeholder>`). It was removed to avoid ref interpolation; the helper already implements and tests `--range`. | `.claude/commands/push-pr-update.md` | skill-audit | Quick | Low |