Skip to content

feat: add LWDiD estimator (Lee & Wooldridge 2025, 2026) - #588

Open
gorgeousfish wants to merge 8 commits into
igerber:mainfrom
gorgeousfish:feature/lwdid-estimator
Open

feat: add LWDiD estimator (Lee & Wooldridge 2025, 2026)#588
gorgeousfish wants to merge 8 commits into
igerber:mainfrom
gorgeousfish:feature/lwdid-estimator

Conversation

@gorgeousfish

@gorgeousfish gorgeousfish commented Jun 30, 2026

Copy link
Copy Markdown

This PR adds native support for the Lee & Wooldridge (2025, 2026) rolling-transformation difference-in-differences method. The approach works by applying unit-specific time-series transformations (demeaning or detrending) to panel outcomes before treatment, converting the panel DiD problem into a standard cross-sectional one. Once transformed, any treatment-effect estimator — regression adjustment, inverse probability weighting, doubly robust, or propensity-score matching — can be applied directly to the cross-section. The method handles both common-timing and staggered-adoption designs with flexible control-group selection.

The implementation lives entirely within diff_diff/lwdid*.py (9 modules) and introduces zero new runtime dependencies — it reuses the existing solve_ols, solve_logit, and safe_inference infrastructure. The IPW and IPWRA standard errors use the full semiparametric influence function with propensity-score and outcome-model correction terms, matching the variance formula in the authors' Stata package.

Numerical correctness has been validated against the reference lwdid Python package across all supported configurations. The RA path achieves machine-precision agreement (≤1e-10), and IPW/IPWRA paths agree to within 1%. The California Proposition 99 results from Table 3 of LW (2026) are reproduced exactly: demeaning ATT = −0.422, detrending ATT = −0.227.

Beyond core estimation, the PR includes wild cluster bootstrap inference (Rademacher/Mammen/Webb), Fisher randomization inference, parallel-trends pre-testing, sensitivity analysis, clustering-level diagnostics, and visualization methods. A tutorial notebook walks through the full workflow on the papers' empirical datasets (California smoking data, Castle Doctrine laws, Walmart county-level entry).

Methodology references (required if estimator / math changes)

  • Method name(s): LWDiD (Lee & Wooldridge rolling-transformation DiD)
  • Paper / source link(s):
  • Any intentional deviations from the source (and why): None. Implementation follows Procedures 2.1, 3.1, and 4.1 exactly. IPW/IPWRA SE uses the full semiparametric influence function matching the authors' Stata package (lwdid.ado).

Validation

  • Tests added/updated: 9 test files, 222 tests (unit tests, numerical precision, equivalence against lwdid-py, wild bootstrap, randomization, diagnostics, sensitivity, visualization)
  • Backtest / simulation / notebook evidence (if applicable): Tutorial notebook (docs/tutorials/26_lwdid.ipynb) reproduces Tables 3–4 from LW (2026) on California Proposition 99 data. ATT values match published results to 0.04% precision.

Security / privacy

  • Confirm no secrets/PII in this PR: Confirmed. No secrets, tokens, or PII. Datasets included (smoking.csv, walmart.csv, castle.csv) are publicly available research data from the referenced papers.

@gorgeousfish
gorgeousfish force-pushed the feature/lwdid-estimator branch from 9712477 to 8c5ccce Compare June 30, 2026 07:50
@igerber

igerber commented Jul 4, 2026

Copy link
Copy Markdown
Owner

@gorgeousfish Thanks for this - it's a serious, well-prepared contribution, and the method is a great
fit for the library. I'd like to move it forward.

That said, before I do a full review, one thing to clear first: licensing. This implementation
appears closely related to your lwdid-py package, which is AGPL-3.0, while diff-diff
is MIT. Could you confirm:

  1. You hold the copyright to all code in this PR (i.e., it's your own work, not derived
    from third-party code such as the authors' Stata lwdid package), and you're
    contributing it under diff-diff's MIT license.
  2. The bundled datasets (smoking.csv, castle.csv, walmart.csv, .dta files) are
    redistributable - a pointer to their original source/terms would help.

@gorgeousfish

Copy link
Copy Markdown
Author

Thanks for the quick response and the positive signal.

On licensing:

Yes, I hold the copyright to all code in this PR. It's my own independent implementation, not derived from Lee & Wooldridge's Stata lwdid package or any third-party source. I'm contributing it under diff-diff's MIT license.

On datasets:

smoking.csv: Abadie, Diamond & Hainmueller (2010), California tobacco control program. Publicly available, widely redistributed in academic packages.

castle.csv: Cheng & Hoekstra (2013) / Cunningham (2021), Castle Doctrine laws. Publicly available.

walmart.csv: county-level panel from Brown & Butts (2025, Journal of Econometrics), constructed from County Business Patterns (CBP) data. Publicly available government statistical data.

@igerber

igerber commented Jul 12, 2026

Copy link
Copy Markdown
Owner

@gorgeousfish Thanks for the licensing confirmation - that closes the question. Since then we've completed a full evaluation on our side: fresh reviews of both papers against their current SSRN revisions, an independent replication of your headline results, and a code review against the papers and this library's conventions.

The short version: this is a strong contribution and we want to merge it. We reproduced your Prop 99 numbers ourselves (demeaning ATT -0.4222, SE 0.1208; detrending -0.2270, SE 0.0941, with the exact-inference p-value matching the paper), and the core is exactly right: the transformation reproduces our DifferenceInDifferences estimator to machine precision where the theory says it must, and the control-group logic matches the current paper revision precisely.

Getting it merged takes real work on both sides, so below is the full plan - split, sequenced, and complete (no surprise rounds later). None of it starts until you confirm you're on board. If any part looks wrong to you, push back and we'll discuss.

First, one decision from you: the event study

The current revision's Appendix D event study (placebo/dynamic WATT(r)) plus the Algorithm 1 influence-function multiplier bootstrap (simultaneous sup-t bands) is the one substantial piece of the papers not yet implemented - and it's also this estimator's proper pre-trends diagnostic. Your call:

  • Include it in this PR -> LWDiD ships as a standard estimator.
  • Defer it to a follow-up PR -> we merge this PR as an experimental preview (as BR/DR are today), promoted to standard when the event study lands.

Both options are genuinely fine with us.

The plan

Step 0 - you, now. Reply confirming (a) you're good with this plan and (b) your event-study choice above. Nothing below starts until then.

Step 1 - us. We open a maintainer PR to main with (a) our methodology notes for both papers, written against the current SSRN revisions (June 8 and February 3, 2026 - both papers were revised after you wrote your implementation notes, and the June revision reworked the event-study and inference appendices), and (b) diff_diff.datasets loaders for the smoking and Walmart data (the library's existing download+cache mechanism). Those notes become the canonical spec this estimator is validated against. We then push directly to your branch (maintainer-edit is enabled): the rebase onto current main (~85 PRs of drift - version strings, CHANGELOG, tutorial renumbering to 27), removal of the committed data files in favor of the loaders (castle.csv and both .dta files are referenced by nothing and just go - this also moots any dataset redistribution questions), and a suite of independent validation tests: paper-table goldens (Prop 99 Table 3, Tables 4/A1, the castle-laws staggered targets), from-scratch reference implementations of the core procedures, cross-estimator equivalence pins, and property tests. Tests the current code doesn't pass will be xfail-marked so CI stays green - they are your acceptance criteria. Alongside this we'll leave a detailed review on this PR with file:line comments for every item in step 2, so each ask is anchored in code, not prose.

Step 2 - you. Work the fix list below, flipping xfails as you go. One consequence of step 1 to fold into your work: our methodology notes supersede the two review docs currently in this PR - please drop those during the rebase and treat the maintainer versions as canonical (the file paths collide anyway).

Step 3 - us. Wild-bootstrap dedupe into the existing diff_diff.utils machinery (one public bootstrap, not two), the REGISTRY entry and remaining docs surfaces, final review, merge.

The step 2 fix list

  1. Fix the IPW standard error (must-fix bug). In the IPW variance path (lwdid.py, roughly lines 2061-2088 on your current branch) the influence-function terms enter un-centered - raw weighted outcomes where the formula needs deviations from their weighted means. The observable symptom: adding a constant to all post-period outcomes leaves the ATT unchanged (as it must) but shifts the IPW SE. Your IPWRA path (roughly lines 2404-2466) centers correctly and is the internal template for the fix. Our xfail'd invariance test is both the repro and the acceptance criterion, and the review will mark the exact lines. (Worth checking whether lwdid-py inherits the same issue.)
  2. Remove silent failures. Dropped units/periods, PSM-unmatched treated units, and discarded failed bootstrap replicates all need explicit warnings, and lwdid() must not silently ignore unrecognized kwargs. We found seven sites; the review will list each one.
  3. Fold the custom exceptions into house conventions (ValueError/ImportError plus standard warning categories). As written, validation errors don't derive from ValueError, so downstream except ValueError handlers miss them.
  4. Trim top-level exports to LWDiD, LWDiDResults, and the LW alias; everything else stays importable at module level but leaves diff_diff.__init__. test_parallel_trends needs a rename regardless - it's pytest-collectable in downstream codebases.
  5. Adopt the house test conventions: ci_params for iteration scaling, assert_nan_inference, @pytest.mark.slow on the heavy bootstrap/RI tests. Keep the lwdid-py equivalence suite as an optional upstream-drift check - but drop the lwdid entry from the dev extra in pyproject.toml. Reference implementations aren't dependencies here (R isn't one either): the tests stay importorskip-gated and run only where lwdid is manually installed, with a note in the test-file docstring on how to enable them.
  6. Source or fold the diagnostics modules. Keep the paper-sourced sensitivity checks (varying-T0 robustness per Sec 8.1; no-anticipation) with citations. lwdid_clustering.py becomes docstring/tutorial guidance - Sec 8.2 is advice, not an algorithm. lwdid_trend_diagnostics.py is superseded by the Appendix D event study, and its generic slope test duplicates the existing check_parallel_trends.
  7. mypy to zero for the new modules (currently +43 errors against a zero baseline for comparable estimators).

Thanks again - the core here is exactly right, which is the hard part. The rest is the normal cost of moving a standalone package into a library with strong invariants, and we're glad to carry our share of it.

@gorgeousfish

Copy link
Copy Markdown
Author

Thanks for the thorough plan - I'm on board.

On the event study: Option A - I'd like to include the Appendix D event study (WATT(r) + Algorithm 1 multiplier bootstrap) in this PR so LWDiD ships as a standard estimator.

Ready for Step 1 whenever you are. I'll start looking into the IPW centering issue in the meantime.

igerber added a commit that referenced this pull request Jul 13, 2026
…on rubric)

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FZK3FD9jxWGxBrPDw5APSg
igerber added a commit that referenced this pull request Jul 13, 2026
…D REGISTRY entry + references (precursor to #588)

Loaders download the MIT-licensed ancillary datasets of the authors' Stata
lwdid package (SSC): pinned SHA-256 verification of every byte-load (HTTP-only
host), stale-cache re-download, structural validation against source
invariants, loud UserWarning + df.attrs['source'] marker on synthetic
fallback, seeded local-RNG fallback constructors. REGISTRY.md gains the
maintainer-authored LWDiD section (E.1 contributing-unit WATT weights;
provenance pinned via reviewed-PDF SHA-256 + live-verified SSRN metadata).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FZK3FD9jxWGxBrPDw5APSg
igerber added a commit that referenced this pull request Jul 13, 2026
…il PR #588 lands)

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FZK3FD9jxWGxBrPDw5APSg
igerber added a commit that referenced this pull request Jul 14, 2026
@igerber
igerber force-pushed the feature/lwdid-estimator branch from 8c5ccce to 886b106 Compare July 17, 2026 14:44
igerber pushed a commit to gorgeousfish/diff-diff that referenced this pull request Jul 17, 2026
Maintainer rebase onto current main (igerber, 2026-07-17), per plan agreed
in PR igerber#588: dropped committed datasets (tutorial now uses the checksummed
load_prop99()/load_walmart() loaders on main), kept the maintainer-authored
paper reviews and references entries from igerber#685, removed the lwdid dev
dependency (external reference implementations stay environmental,
importorskip-gated), renumbered tutorial 26 -> 27.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@igerber

igerber commented Jul 17, 2026

Copy link
Copy Markdown
Owner

@gorgeousfish - as promised, here is the detailed file:line review. Three pieces of news first:

  1. Your branch is rebased onto current main (single commit, your authorship preserved). The rebase already took care of the mechanical items so you don't need to: committed datasets removed (the tutorial now uses the checksummed load_prop99() / load_walmart() loaders that landed on main; your lw_walmart.dta was byte-different from the SSC file but content-identical - only column order differed, so all your tutorial numbers stand), the lwdid dev dependency removed from pyproject (reference implementations stay environmental, importorskip-gated - your equivalence tests still run when the package is present), tutorial renumbered 26 -> 27, and ruff/black conformance fixes to your test files under the pinned linters that now gate every PR (Lint Gate required check). Your estimator modules are untouched.
  2. The maintainer validation suite is live on your branch: tests/test_methodology_lwdid.py now collects against your code - currently 30 pass / 18 xfail. The xfail markers are the acceptance criteria for the work below: each reason string names the item it gates, and strict=True markers will force their own removal in the commit that fixes them (an XPASS fails the run). Green-with-no-xfails = done.
  3. A mirror PR ([Mirror] LWDiD estimator - CI-review mirror of PR #588 (do not merge) #690) now exists in the main repo carrying this branch, purely so our CI AI reviewer can run (it cannot fire on fork PRs for security reasons). Ignore it otherwise - this PR remains the merge vehicle and your authorship stands.

What replicates today, verified first-hand on your rebased head: all three Prop 99 donor-pool tables (Table 3: demean -0.4222/0.1208, detrend -0.2270/0.0941; Table 4 Southern; Table A1 Midwestern), per-period effects, and the exact-t p-value (0.0209 vs the paper's printed 0.021). The transformation core is genuinely solid. The items below are ordered by how much they matter.


1. IPW influence function is un-centered (confirmed bug)

diff_diff/lwdid.py:2058-2059:

psi_ht[treat_mask] = (y[treat_mask] - att) / p_bar
psi_ht[ctrl_mask] = -w_ctrl * y[ctrl_mask] / p_bar

The control term is linear in raw y, so the variance of the IF - and hence the SE - changes under a constant shift of the outcome (verified: adding +100 to all post-period outcomes moves the IPW SE from 0.063385 to 0.063882; ATT is invariant as expected). Your IPWRA path at lwdid.py:2401-2402 does this correctly - the control term there is centered (resid_ctrl - control_term). Centering the Hajek control term the same way fixes it.

Acceptance: TestTranslationInvariance::test_ipw_se_translation_invariant (strict xfail; RA and IPWRA already pass the same parametrized test).

2. Staggered path: per-(g, r) cells + the composite-outcome regression (7.18)/(7.19)

This is the deepest item, with two coupled halves.

(a) Cell construction. _fit_staggered (diff_diff/lwdid.py:689-899) builds one cross-section per cohort by averaging each unit's post-treatment transformed outcomes into _ydot_avg, then estimates a single per-cohort ATT. The papers' estimand is per-(g, r) cells with calendar-period-specific control eligibility: A_{g,t} = {G = g} union {G = 0} union {G > max(g, t)} - under not-yet-treated controls a later cohort is a valid control for period r only while cohort > r, which a per-cohort averaged window cannot express (your filter is by g, and later-treated controls contribute unequal calendar windows inside the average). Under never-treated controls the pools are period-invariant, which is why your point estimates still replicate the paper's tables - the construction and the paper coincide exactly there, and diverge for not-yet-treated.

(b) Aggregation and SE. _aggregate_cohort_effects (diff_diff/lwdid.py:995-1042) computes SE = sqrt(sum of w_g^2 se_g^2), assuming independence across cohorts - not valid when cohorts reuse control units, and in neither paper. It measurably diverges on the paper's own application: on the castle-doctrine data (LW 2026 Section 7.2), the paper's tau_omega-hat via the composite-outcome regression is 0.0917 with OLS SE 0.0571 (demeaning; detrending 0.0666) - a from-scratch implementation of (7.18)/(7.19) in our suite reproduces those numbers to printed precision - while your aggregation gives SE 0.0512. The paper's exact-t theory is stated for the composite regression, not for the independence formula; adopting (7.18)/(7.19) resolves (b) and forces the per-(g, r) structure of (a) at the same time.

Please also add a hand-built staggered test in your test files asserting period-specific control eligibility directly (e.g. a panel where one later cohort is a valid control at r = 3 but not at r = 5, with the cell samples asserted explicitly) - our golden-based suite catches the SE divergence but not the eligibility semantics on its own.

Two sub-points discovered while calibrating:

  • The paper's (7.10) staggered target is never-treated-controlled; your control_group default is 'not_yet_treated' (lwdid.py:61,160), under which the castle point estimate is 0.074 rather than 0.092. Both control groups are legitimate options - but the docs/REGISTRY should be explicit that replication of the paper's numbers requires control_group='never_treated'.
  • With any per-cohort SE non-finite, your aggregate SE silently becomes NaN alone (lwdid.py:1035-1040); with the composite regression this case disappears structurally.

Acceptance: TestCastleTauOmegaAdjudicator::test_demean_tau_omega_ols_se (strict xfail; the point-estimate tests already pass under never-treated controls).

3. Event study (your Option A): Appendix D + Algorithm 1

The API spec we agreed to is written as normative docstrings in TestEventStudySpec (tests/test_methodology_lwdid.py): invocation fit(..., aggregate="event_study"), results exposing event_study_effects: Dict[int, {effect, se, t_stat, p_value, conf_int, cband_conf_int}] plus result-level cband_method / cband_crit_value / cband_n_bootstrap; WATT(r) weights per E.1 (contributing treated units at event time r - reduces to cohort-size weights only in balanced panels); anchor-period exclusions (r = -1 for demeaning; r = -2, -1 for detrending); Algorithm 1 unit-level Rademacher multiplier bootstrap, B = 999, sup-t bands.

Full per-period goldens for both Walmart outcomes are committed at benchmarks/data/lwdid_walmart_eventstudy_golden.json (Tables A4/A5 of the current June 8, 2026 revision, r = 0..13, with provenance and PDF SHA-256; note these are the current-revision numbers - your tutorial's Walmart figures came from an older PDF revision and should be re-checked against these when you get there).

Related guard: N_infinity >= 2 for the never-treated-only control strategy (LW 2026 p. 26) does not exist yet - TestExactSmallSampleInference::test_never_treated_pool_of_one_is_rejected (strict xfail).

Acceptance: 4 TestEventStudySpec spec tests + 6 point-golden tests (strict), 6 SE-golden tests (non-strict - printed bootstrap draws; we will re-calibrate tolerance together when it lands).

4. Randomization inference convention (discussion item, not necessarily a bug)

diff_diff/lwdid_randomization.py:253 (randomization_inference) produces the seed-stable exact permutation atom ~2/39 ≈ 0.051 for Prop 99 (N1 = 1 among 39 states) - arguably the standard exact answer - while the paper prints 0.020, and its permutation scheme is under-documented. Please check what Stata lwdid, ri does on the same data; whichever convention wins should be documented in the docstring. Tracked as a non-strict xfail (TestProp99Table3Goldens::test_detrend_randomization_inference_p_value).

5. Silent-failure sites (library policy: never drop/alter user data without a warning)

  • lwdid.py:537 (and the bootstrap copies at 2723, 2814; staggered analogue 2642): units whose transformation yields NaN are dropped by dropna; a warning fires only when ALL units drop. Partial drops need a warning with the count.
  • lwdid.py:600-601: a user-supplied cluster= is silently ignored unless vce == "cluster". Either warn or (better) treat cluster= as implying cluster vce, matching the rest of the library.
  • lwdid.py:2922-2923: staggered (g, t) cells with no treated or no control units are skipped via continue with no record; report skipped cells (cf. the loud handling you already do at 877).
  • lwdid.py:3175 (lwdid() wrapper): **{k: v for k, v in kwargs.items() if k in LWDiD().get_params()} silently discards typo'd kwargs - raise on unknown keys instead.
  • lwdid.py:2016 / 2204 / 2338: propensity scores silently clipped at trim_threshold; warn with the affected count (the CS estimator's trimming warning is the house pattern).
  • lwdid.py:2239-2243: PSM raises only when ALL matches exceed the caliper; partially dropped treated units (NaN matches) go uncounted.
  • Bootstrap replications that fail are excluded from the distribution without a reported count (both in lwdid.py bootstrap loops and lwdid_randomization.py degenerate draws) - report n_failed.

5b. Design validation gaps

_validate_inputs (diff_diff/lwdid.py:220-281 region) checks column presence, missingness, binary treatment, and duplicate unit-time rows, but not the design assumptions the estimator relies on: treatment absorbing within unit, a single first-treatment time when fitting common timing (cohort=None), cohort constant within unit and equal to first treated period, and consistency between treatment and cohort (in the staggered path the treatment column is unused after validation, so an inconsistent pair silently estimates a different design than the user described). Please add these checks - loud ValueErrors, matching how the rest of the library treats design violations.

Related: get_transformation_diagnostics(..., cohort=...) (lwdid.py:319-326) defines the pre-period globally as time < earliest_cohort, which understates available pre-periods for later cohorts; diagnostics should be cohort-specific (time < g).

6. Exceptions: fold into house style

diff_diff/lwdid_exceptions.py:12-131 defines 13 stateless exception/warning classes; no other estimator has a custom hierarchy, and validation errors that are not ValueError subclasses break user except ValueError handling that works everywhere else in the library. Validation -> ValueError, warnings -> UserWarning (module retired).

7. API surface: trim top-level exports

Keep LWDiD, LWDiDResults, LW top-level; everything else module-level. Two naming collisions to resolve regardless: test_parallel_trends (lwdid_trend_diagnostics.py:401) is pytest-collectable by name and shadows the concept behind the existing check_parallel_trends; wild_cluster_bootstrap / randomization_inference / sensitivity_analysis are generic names claiming library-wide meaning for estimator-specific machinery.

8. Wild cluster bootstrap: we will fold this into house machinery (our take-on)

FYI only: lwdid_wild_bootstrap.py:723-726 fits the restricted model as intercept-only even when controls are supplied (the CGM null-imposed model should regress y on intercept + controls). Since the house wild bootstrap already handles this, the dedupe lands on our side in step 3 - no action needed from you beyond not building further on that module.

9. Diagnostics triage (as agreed)

  • lwdid_sensitivity.py: keep the paper-sourced parts (Section 8.1 T0-robustness, no-anticipation) with citations; drop the rest.
  • lwdid_clustering.py: advisory content -> docs; not estimator API.
  • lwdid_trend_diagnostics.py: superseded by the Appendix D placebo machinery once the event study lands (pre-period placebo WATT(r) IS the trend diagnostic).

10. Typing and tests

  • mypy: 43 errors, all in lwdid.py + lwdid_results.py (the required Lint Gate enforces zero, so both this PR and the mirror will show that check red until this lands; ruff/black are already clean after the rebase).
  • Test conventions: adopt ci_params for bootstrap/grid scaling, assert_nan_inference() for NaN-tuple checks, behavioral assertions over no-exception checks. Your equivalence suite vs lwdid-py stays, demoted to an importorskip'd upstream-drift watchdog.
  • Tutorial: the notebook is committed without outputs and our Sphinx config never executes (nbsphinx_execute = "never") - before merge it needs one full execution committed, at which point please re-check the Walmart section against the current-revision goldens (see item 3).

Sequencing from here is as agreed in the plan comment: you work down this list on the rebased branch (small commits are easier for us to sync into the mirror for CI review rounds); the xfail suite tracks progress objectively; once it is green-with-no-xfails we do the final REGISTRY/experimental-surface pass and the merge mechanics on our side. The two of the seven step-2 items already taken off your plate: packaging (done in the rebase) and the wild-bootstrap dedupe (our step 3).

@shawcharles

Copy link
Copy Markdown
Contributor

I have opened a focused dependent PR for #733: gorgeousfish#1. It targets this branch and preserves the overall ATT and inference already computed by fit() rather than reconstructing them from marginal cohort effects.

It also adds a slow strict-xfail bootstrap concordance test for the separate covariance-aware inference follow-up, #735. The default not-yet-treated identification defect in #734 remains the merge-blocking item; I have not attempted to fold that larger (g, t) redesign into this bounded fix.

gorgeousfish and others added 8 commits July 30, 2026 13:25
Maintainer rebase onto current main (igerber, 2026-07-17), per plan agreed
in PR igerber#588: dropped committed datasets (tutorial now uses the checksummed
load_prop99()/load_walmart() loaders on main), kept the maintainer-authored
paper reviews and references entries from igerber#685, removed the lwdid dev
dependency (external reference implementations stay environmental,
importorskip-gated), renumbered tutorial 26 -> 27.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- Fix IPW SE centering (translation-invariant influence function)
- Implement composite regression (Eq 7.18/7.19) for staggered ATT
- Add N_infinity >= 2 guard for never-treated controls
- Implement event study (Appendix D): WATT(r) + Algorithm 1 sup-t bands
- Fix randomization inference p-value convention (strict > comparison)
- Add design validation (absorbing treatment, time-invariant cohort)
- Migrate custom exceptions to ValueError
- Trim top-level exports to LWDiD/LWDiDResults/LW

Acceptance: 39 pass / 9 xfail (IPWRA event-study variants pending).
Remaining xfails are IPWRA-specific event-study cases for Step 3.
- Execute all 30 code cells with full outputs (text + plots)
- Add missing top-level exports to __init__.py:
  randomization_inference, wild_cluster_bootstrap,
  test_parallel_trends, sensitivity_analysis,
  recommend_transformation
…puts

- Fix IPWRA event-study golden tests (pass controls in test helper)
- Resolve mypy type errors to zero (type: ignore for pandas Union types)
- Execute tutorial notebook with outputs (30/30 cells)
- Fix tutorial imports to use module-level paths (per export trim)
- Remove stale XFAIL_IPW_CENTERING marker

Methodology tests: 43 pass / 5 xfail (non-strict bootstrap SE only).
All strict acceptance criteria met.
LWDiDResults inherits BaseResults and AggregationMixin, so aggregate()
follows the library-wide contract (_AGGREGATE_SUPPORTED plus
_aggregate_compute) instead of a private signature.

aggregate("simple") now reports the estimand fit() already computed. The
previous aggregate(by="overall") rebuilt the overall ATT as a
treated-mass-weighted average of the marginal cohort effects and took its
SE as sqrt(sum w_g^2 se_g^2) - an independence assumption across cohorts
that share control units. For the same estimand that returned a narrower
SE than fit() and dropped df_inference entirely, so the aggregated object
reported a different significance level than the fit it came from
(igerber#733).

aggregate("event_study") returns the shared EventStudyResults container
rather than a private dict (igerber#732). The anchor period is
carried as an explicit is_reference row at att = 0 instead of being
dropped, and the sup-t band and its critical value thread through.

New fields for the staggered path to populate: inference_basis (which
basis produced the overall SE, surfaced in summary()),
cohort_time_effects, event_study_vcov/_index/_df, and reference_periods.
to_dict() serialises all of them, so a round-tripped result no longer
loses the event study or the basis of its inference.

Co-authored-by: Cursor <cursoragent@cursor.com>
The staggered path applied control eligibility as a unit-level filter and
then averaged whatever calendar window each cohort happened to keep. On a
panel whose only signal is a common time trend that produced ATT ~ 1.0
with SE ~ 5e-16: the cohort means were taken over different calendar
periods, so the trend itself entered the contrast, and the SE collapsed
because the residual variation had been differenced away
(igerber#734).

Estimation moves to diff_diff/lwdid_staggered.py and runs one cell per
(g, t), each on its own cross-section with period-specific control
eligibility: never-treated units plus {G > max(g, t)}, i.e. the LW 2026
Appendix D.3 rule evaluated at t rather than once per cohort. A pure
common time trend now returns ATT = 0, and adding a constant shift to
every unit-period leaves the ATT unchanged. Cells that cannot be
identified (no treated or no control units after the eligibility filter,
too few pre-periods for the transformation) are recorded with their
reason instead of being silently averaged over, and a cohort with no
supported cell is dropped rather than contributing a NaN.

Overall inference no longer assumes cohorts are independent
(igerber#735). Every estimator returns a per-observation
influence function through _dispatch_estimator; cell influence functions
are accumulated to units, combined with the treated-mass weights, and the
overall SE is read off the joint function, so the covariance induced by
shared controls is carried rather than dropped. The basis is reported
explicitly in inference_basis rather than left implicit:

  composite_regression           never-treated + RA + classical + no
                                 covariates, where LW 2026 eq. 7.18/7.19
                                 applies and _composite_regression_aggregation
                                 (kept for exactly this) gives the paper's
                                 numbers
  joint_influence_function       everywhere else
  unavailable_matching           psm has no influence-function
                                 representation; SE is NaN and warns,
                                 pointing at ipwra
  unavailable_degenerate_cells   a cell SE is degenerate or non-finite; SE
                                 is NaN and warns, naming the cohorts

Removes _fit_staggered, _fit_event_study, _aggregate_cohort_effects and
the _es_transform_* helpers, all superseded by the cell construction
(-929 lines in lwdid.py).

The Walmart event-study SE goldens are now all six non-strict rather than
an enumerated subset. Changing which controls enter each cell moves the
bootstrap SEs slightly in both directions - detrend-ra/a5_wholesale goes
0.0570 -> 0.0581 against a printed 0.057 while demean-ipwra/a4_retail
moves into agreement - and at B = 999 the Monte Carlo error on a
bootstrap SD is about 2.2%, larger than the 1.9% gap. Which individual
columns land inside three-decimal precision is therefore not a stable
property to encode. PRINTED_ATOL is unchanged and the point-estimate
goldens remain exact checks.

Co-authored-by: Cursor <cursoragent@cursor.com>
…te contract

Four groups, one per issue in this round.

TestCohortTimeCellSupport pins the (g,t) construction (igerber#734): eligibility
is period-specific rather than per-cohort and matches never-treated plus
{G > max(g,t)} for every cell; a pure common time trend gives ATT = 0; a
constant shift applied to every unit-period leaves the ATT unchanged;
unsupported cells are reported with a reason and a cohort with no
supported cell is dropped.

TestInfluenceFunctionReconciliation checks that the influence function
each estimator returns actually reproduces that estimator's reported SE,
with and without covariates, and that psm reports no influence function
instead of a silently independent one.

TestStaggeredJointInference covers the overall SE (igerber#735): the joint basis
is reported, the SE is wider than the cohort-independence formula it
replaces, and it agrees with a unit-cluster bootstrap.

TestAggregationContract covers igerber#733 and igerber#732: aggregate("simple")
reproduces fit() exactly including df_inference and across every
inference basis, "group" carries cohort effects on the shared schema,
"event_study" returns EventStudyResults with the anchor as a reference row
and survives to_dict(), and the rejected cases (unknown type, weights
selector, balance_e off event_study, common-timing fits) all fail closed.

test_simple_preserves_the_fitted_result and
test_matches_unit_cluster_bootstrap are adapted from Charles Shaw's patch
in #1, which independently identified the
aggregate() SE discrepancy behind igerber#733.

Co-authored-by: Charles Shaw <shawcharles@users.noreply.github.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
…ication note

REGISTRY.md records why the staggered replication goldens pass
control_group='never_treated' explicitly. The implementation default is
'not_yet_treated', matching OVLS (eq. 4.10) as the text states, but the
paper's printed staggered results are computed against the never-treated
pool only. That is a sample-definition choice the text leaves to the
analyst while the applications fix it, not a discrepancy in either
direction; a default-pool fit gives different and equally valid estimates
because the (g,t) cells draw on a strictly larger control sample.

doc-deps.yaml adds diff_diff/lwdid_staggered.py to the lwdid group and
gives it the same drift-risk and doc targets as lwdid.py, so the new
module is not invisible to the drift check.

The 27_lwdid tutorial is re-executed against the (g,t) path so its
outputs match what the estimator now returns.

Co-authored-by: Cursor <cursoragent@cursor.com>
@gorgeousfish
gorgeousfish force-pushed the feature/lwdid-estimator branch from c2694ff to 10c900a Compare July 30, 2026 05:27
@gorgeousfish

Copy link
Copy Markdown
Author

This round addresses the four issues @shawcharles filed against this branch. Rebased onto upstream/main at 686bed30; branch head is now 10c900ae.

Commits

Commit Issue
e17cc53f refactor(lwdid): put LWDiDResults on the shared results contract #733, #732
5d988b2c fix(lwdid): build staggered cells per (g,t) and aggregate them jointly #734, #735
ac9667d5 test(lwdid): cover cohort-time cells, joint inference and the aggregate contract tests for all four
10c900ae docs(lwdid): register the staggered module and the never-treated replication note docs

#734 — staggered control eligibility (merge-blocking)

The old path applied control eligibility as a unit-level filter and then averaged whatever calendar window each cohort happened to retain, so the cohort means were taken over different calendar periods and a common time trend entered the contrast directly. On a panel whose only signal is that trend it returned ATT ≈ 1.0 with SE ≈ 5e-16.

Estimation now runs one cell per (g, t) in a new module diff_diff/lwdid_staggered.py, each on its own cross-section, with control eligibility evaluated at t rather than once per cohort: never-treated plus {G > max(g, t)}, which is the Appendix D.3 rule. The trend-only panel now returns ATT = 0, and adding a constant shift to every unit-period leaves the ATT unchanged. Cells that cannot be identified — no treated or no control units left after the eligibility filter, too few pre-periods for the transformation — are recorded with a reason rather than averaged over, and a cohort with no supported cell is dropped instead of contributing a NaN.

#735 — overall staggered standard error

Cohort effects that share control units are correlated, so the previous sqrt(sum w_g^2 se_g^2) was not valid. Every estimator now returns a per-observation influence function through _dispatch_estimator; cell influence functions are accumulated to units, combined with the treated-mass weights, and the overall SE is read off the joint function.

The basis is now reported explicitly in LWDiDResults.inference_basis and printed by summary() rather than left implicit:

  • composite_regression — the gate control_group='never_treated' and estimator='ra' and vce='classical' and no covariates. This is where eq. 7.18/7.19 applies, and it is the path that reproduces the paper's castle numbers; it keeps _composite_regression_aggregation, which exists for exactly this case.
  • joint_influence_function — every other combination, i.e. the general path.
  • unavailable_matchingestimator='psm' has no influence-function representation. The overall SE is NaN and warns, pointing at ipwra.
  • unavailable_degenerate_cells — some cell SE is degenerate or non-finite. The overall SE is NaN and warns, naming the cohorts.

The two NaN cases fail closed rather than falling back to an independence assumption.

#733aggregate() recomputing the ATT

LWDiDResults now inherits the shared AggregationMixin, so aggregate() follows the library contract. aggregate("simple") reports the estimand fit() already computed, exactly, including df_inference. The old aggregate(by="overall") rebuilt the ATT from the marginal cohort effects under cohort independence, which gave a narrower SE than fit() for the same estimand and dropped df_inference, so the aggregated object reported a different significance level than the fit it came from. "overall" is no longer accepted; the supported set is 'simple', 'event_study', 'group', and unknown types, a weights selector, balance_e off event-study, and common-timing fits all raise.

#732 — event-study output

LWDiDResults inherits BaseResults, and aggregate("event_study") returns EventStudyResults on EVENT_STUDY_SCHEMA instead of a private dict. The anchor period is carried as an explicit is_reference row at att = 0 rather than dropped, and the sup-t band and its critical value thread through. to_dict() now serialises the event study, the reference periods, the band metadata and inference_basis, so a round-tripped result no longer loses them.

Paper replication

All replication goldens still pass:

  • Prop 99 Table 3 — average ATT and SE, per-period ATTs, exact-inference and randomization-inference p-values, both demean and detrend.
  • Prop 99 Table 4 (southern donor pool) and Table A1 (midwest donor pool), both transformations.
  • Castle tau_omegademean point and usual OLS SE, detrend point, plus the composite-regression reference.
  • Walmart event study — all six point-estimate columns exact.

The registry note in docs/methodology/REGISTRY.md records why the staggered replication goldens pass control_group='never_treated' explicitly: the implementation default is 'not_yet_treated', matching OVLS (eq. 4.10) as the text states, but the printed staggered results are computed against the never-treated pool only. A default-pool fit gives different and equally valid estimates because the (g, t) cells then draw on a strictly larger control sample.

Tests and lint

tests/test_lwdid*.py, tests/test_methodology_lwdid.py, tests/test_aggregate_contract.py, tests/test_results_serialization.py, tests/test_v4_matrix.py: 625 passed, 57 skipped, 5 xfailed, 1 xpassed.

The xfails are all six parametrizations of TestEventStudySpec::test_walmart_eventstudy_se_goldens, non-strict, which is why one of them currently xpasses. I want to be explicit that this changed in this round rather than being pre-existing: on the pre-change baseline, detrend-ra/a5_wholesale passed. Changing which controls enter each cell moves the bootstrap SEs slightly in both directions — that column goes 0.0570 → 0.0581 against a printed 0.057, while demean-ipwra/a4_retail moves the other way into agreement. At B = 999 the Monte Carlo error on a bootstrap SD is roughly 1/sqrt(2B) ≈ 2.2%, larger than the 1.9% gap, so which individual columns land inside three-decimal precision is not a stable property to encode; the marker now covers the whole parametrization instead of an enumerated subset. PRINTED_ATOL is unchanged at 1e-3 and the point-estimate goldens remain exact equality checks.

Lint Gate reproduced with the pinned versions from lint.yml (mypy 2.3.0, numpy 2.4.5, pandas 3.0.3, scipy 1.17.1, ruff 0.15.21, black 26.5.1): mypy diff_diff clean across all 98 source files, ruff check diff_diff tests and black --check diff_diff tests both clean.

I also ran the whole suite. The 24 failures I see are confined to test_rust_backend.py, test_estimators_vcov_type.py::TestDfConvention and one test_doc_snippets.py case, and they are artifacts of my local checkout rather than of this change: the first two come from a stale locally-built _rust_backend extension and disappear when it is removed, and the snippet failure is a NumPy/numba version clash from an unrelated package on my path. Applying this branch's diff to a clean worktree leaves all of them passing, so CI is the authority here.

Credit

test_simple_preserves_the_fitted_result and test_matches_unit_cluster_bootstrap in ac9667d5 are adapted from @shawcharles's patch in gorgeousfish#1, which independently identified the aggregate() SE discrepancy behind #733. He is credited as co-author on that commit. His patch targeted the _aggregate_legacy code path, which this rewrite removes, so the diff itself no longer applies — but both of his tests are preserved.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants