feat(3.9): converge clustered CR1 on K_reference - D1+D2 fix (M-126) - #737
Conversation
PR B of the 3.9 variance-consolidation program: the clustered hc1 CR1
finite-sample factor (G/(G-1))*((n_eff-1)/(n_eff-k)) now uses
K_reference = explicit columns
+ (1 for the absorbed constant when X carries no intercept col)
+ rank(absorbed FE not nested in the cluster | nested ones)
threaded as a NEW keyword-only signed `cluster_k_adjustment: int = 0` on
solve_ols / compute_robust_vcov / LinearRegression.fit / wild_bootstrap_se,
computed by the new utils helpers (absorbed_fe_cr1_k_increment,
cluster_nested_fe_dims, shared factorizations with absorbed_fe_rank).
Fixes two defects from the measured variance-conventions inventory:
- D2 (anti-conservative): absorbed FE not nested in the cluster were never
counted - clustered SEs understated by 0.10% (n=2500) / 1.30% (n=200) /
5.51% (n=60). WooldridgeDiD now matches Stata jwdid/reghdfe at machine
precision on all committed mpdta arms AND the new G~20..500 subsample
ladder golden (historical ratios 1.0280@G=20 down to 1.0010@G=500, all
now 1.0 at spreads ~1e-15..1e-14; the ladder doubles as the K-accounting
probe: reghdfe df_a == increment - 1 per rung). SunAbraham matches
fixest::sunab at 3.1e-14 rel (was ~1-2%; closes the DEFERRED row). TWFE
fixest cluster gates tightened from a 0.5% band to exact / rtol 1e-9.
- D1 (the 10.35% idiom split): absorb=[unit,time] vs fixed_effects=[unit,
time] on the same clustered model returned SEs differing by exactly
sqrt((360-2)/(360-66)); the full-dummy lane now subtracts the
cluster-nested dummy RANK, so both documented-equivalent spellings return
the identical externally-anchored SE (holds under collinearity drops and
on disconnected panels - kernel k is the design rank).
Wired surfaces: DiD/MPD/TWFE (absorb increment; fixed_effects negative
nested rank; MPD's built-in period dummies are its time-FE block and follow
the nested convention), SunAbraham, WooldridgeDiD OLS within, ImputationDiD
pretrends lead regression, LPDiD nested dummy blocks (with an
InvalidClusterKAdjustment re-raise so the broad except cannot swallow the
contract), and the WCB corr constant (cancels in |t*| vs |t0|: bootstrap
p-values invariant; reported SE aligns with the analytical convention).
StackedDiD (clubSandwich CR1S by construction) and Wooldridge cohort_trends
full-dummy keep their documented conventions bit-identically. Rust lanes
apply the correction as an exact None-preserving scalar rescale of the
finished vcov in the Python wrappers (zero Rust changes, <= 1 ulp;
zero-adjustment surfaces bit-identical).
Contracts: front-door InvalidClusterKAdjustment(ValueError) validation on
every route (type half checked unconditionally at entry - classical/survey
fits and degenerate WCB returns reject non-ints too); fail-closed all-NaN
vcov on all three saturation sides (n_eff-k <= 0, n_eff-k_inf <= 0,
k_inf <= 0) on both backends, pinned at the exact boundaries with one-step
finite recovery; survey fits are the documented inert VALUE exception.
External anchors beyond Stata/fixest parity: the exact non-nested RANK term
is anchored on a DISCONNECTED two-way panel against BOTH reghdfe (2.8e-17)
and fixest ssc(K.fixef="full", K.exact=TRUE) (5.6e-16) via two new goldens
(fixest_cr1_nonnested_golden.json, reghdfe_kref_golden.json - deterministic
RNG-free DGP); no external reference implements nested-drop +
exact-remainder (fixest's K.exact composes incoherently with its nested
drop; reghdfe's pairwise correction skips nested-dropped dims), so that
one-df deviation is pinned exactly as sqrt((n-10)/(n-11)) against both and
documented as a labeled REGISTRY deviation. N>=3 absorbed dims inherit the
documented D3 sum(levels)-N+1 approximation (limitation pinned; TODO row).
Tail df (df_, dof_vec, inference_df, df_convention) deliberately unchanged
- PR C scope. Ledger: docs/v4-deprecations.yaml M-126 (+ v4-matrix snapshot
105); REGISTRY/variance-conventions/CHANGELOG/benchmark READMEs rewritten
to the converged convention; DEFERRED rows 53+54 closed; TODO ladder row
closed into the benchmarks.rst refresh row; perf note re-measured
(increment 3.2 ms/call at 186k rows, tracked).
Overall assessment✅ Looks good No unmitigated P0 or P1 findings. Executive summary
Methodology
The implementation itself follows the registered formula at Code QualityNo findings. Performance
Maintainability
Tech DebtNo untracked findings. The N≥3 rank approximation, benchmark refresh, and factorization optimization are recorded in SecurityNo findings; no secrets, unsafe deserialization, or new external-input execution paths observed. Documentation/Tests
Coverage added for weighted/rank-deficient forwarding, exact saturation boundaries, Rust parity, wild-bootstrap invariance, and external Stata/fixest goldens is proportionate to the methodological risk. |
…(review round 1, 2x P2) - The _absorbed_fe_vcov_scale gate comments in linalg.py and the MPD twin in estimators.py still said "clustered SEs keep k_visible (fixest ssc nested-FE convention)"; clustered CR1 now carries the K_reference accounting through cluster_k_adjustment inside the kernel, and the two paths are mutually exclusive on cluster_ids - the comments now say so. - test_variance_conventions.py module docstring updated: cr1_k records K_reference counts (visible + the signed adjustment), and the remaining documented-defect rows are the tail-df conventions (PR C), not the closed anti-conservative k accounting.
|
🔁 AI review rerun (requested by @igerber) Head SHA: Overall assessment✅ Looks good No unmitigated P0 or P1 findings. Executive summary
Methodology
The implemented formula and fail-closed boundaries are consistent with the Registry at Code QualityNo findings. Performance
MaintainabilityNo findings. The prior stale-comment issue is resolved at Tech Debt
SecurityNo findings; no secrets or unsafe execution paths detected. Documentation/TestsNo findings. The prior audit-header issue is resolved at |
…ticmethod @pytest.fixture stacked on @staticmethod collects fine on Python >= 3.10 but crashes pytest collection on 3.9 (staticmethod has no __name__ there), which is the library floor and exactly one CI matrix leg - the form passed on every other leg and locally (3.14). The module-level fixture form satisfies both constraints: no PytestRemovedIn10Warning (the reason the staticmethod form was introduced) and 3.9-compatible collection.
|
🔁 AI review rerun (requested by @igerber) Head SHA: Overall assessment✅ Looks good No unmitigated P0 or P1 findings. Executive summary
Methodology
The implemented factor and fail-closed boundaries agree with the Registry at Code QualityNo findings. Performance
MaintainabilityNo findings. Parameter propagation is complete across the affected estimator and bootstrap paths, and the prior stale comments are resolved. Tech Debt
SecurityNo findings. No secrets, unsafe execution paths, or security-sensitive changes were identified. Documentation/TestsNo findings. External parity, weighted and rank-deficient paths, saturation boundaries, Rust scaling, and wild-bootstrap invariance are covered. The module-level ladder fixture at |
1 participant
Summary
K_reference— D1+D2 of the 3.9 variance-consolidation program ([M-126]): the factor'sknow counts explicit columns + the absorbed constant (no-intercept designs) + the rank of absorbed FE not nested in the cluster, threaded as a new keyword-only signedcluster_k_adjustmentthroughsolve_ols/compute_robust_vcov/LinearRegression.fit/wild_bootstrap_se(additive, default 0).jwdid/reghdfe at machine precision on all committed arms and a new G≈20..500 subsample ladder golden (historical ratios 1.0280→1.0010, all now 1.0 at spreads ~1e-15..1e-14); SunAbraham matchesfixest::sunabat 3.1e-14 relative (was ~1-2%); TWFE fixest cluster gates tightened from a 0.5% band to exact / rtol 1e-9.absorb=andfixed_effects=spellings of the same clustered model now return the identical externally-anchored SE (the full-dummy lane subtracts the cluster-nested dummy rank; equivalence is rank-based, so it holds under collinearity drops and on disconnected panels).InvalidClusterKAdjustmentre-raise so the broad except cannot swallow the contract), and the wild-cluster-bootstrap reported SE (the correction cancels in the studentized statistic — bootstrap p-values invariant). StackedDiD (clubSandwich CR1S) and Wooldridgecohort_trendsfull-dummy keep their documented conventions bit-identically; Rust lanes apply an exact scalar rescale (zero Rust changes, ≤1 ulp, zero-adjustment surfaces bit-identical).n−k ≤ 0,n−k_inf ≤ 0,k_inf ≤ 0) on both backends, pinned at exact boundaries; front-door validation on every route with the type contract checked unconditionally at entry.ssc(K.fixef="full", K.exact=TRUE)(5.6e-16); no external reference implements nested-drop + exact-remainder, so that one-df deviation is pinned exactly (sqrt((n−10)/(n−11))) against both and documented as a labeled REGISTRY deviation.Methodology references (required if estimator / math changes)
jwdid, Rios-Avila); R fixest 0.14.2ssc()(K.fixefnested convention,K.exact);docs/methodology/variance-conventions.md(D1/D2 derivation + measured inventory);docs/methodology/REGISTRY.mdvariance-family notes.fixed_effects=subtracts nested explicit-dummy rank where literal R explicit-dummy regressions count all columns — the deliberate consistency choice so two documented-equivalent kwargs return identical SEs (labeled REGISTRY deviation; anchored via the absorb side's external parity + the equality gate). (2) The non-nested term is the exact rank given the nested set; on disconnected nested designs this sits one df below both references, which only approximate that composition (labeled REGISTRY deviation, pinned exactly against both). (3) N≥3 absorbed dims inherit the documented D3sum(levels)−N+1approximation (limitation pinned; TODO row). Tail-df conventions deliberately unchanged (PR C).Validation
tests/test_variance_conventions.py(re-pinned audit matrix + 4 new rows, K_reference convergence/boundary/WCB tests, two golden-parity classes, N≥3 limitation pin),tests/test_linalg.py(seam contracts: validation on every route, exact boundaries, weighted/rank-deficient forwarding, Rust parity, signature contract),tests/test_etwfe_cs_stata_parity.py(ratio gates → 1.0 + the subsample-ladder class),tests/test_fixest_did_twfe_parity.py(tightened per-arm),tests/test_methodology_twfe.py/test_methodology_sun_abraham.py(bands → machine-precision pins),tests/test_wooldridge.py(baseline re-capture),tests/test_estimators_vcov_type.py(divergence → convergence),tests/test_v4_matrix.py(M-126 snapshot),tests/test_methodology_wooldridge.py(prose).etwfe_cs_stata_golden.json(ladderblock; pre-existing arms bit-identical),fixest_cr1_nonnested_golden.json,reghdfe_kref_golden.json(deterministic RNG-free DGP).Security / privacy