Skip to content

Target-parity contract for populace-us: gate + wire missing SSA SSI families#404

Merged
MaxGhenis merged 4 commits into
mainfrom
target-parity-contract
Jul 13, 2026
Merged

Target-parity contract for populace-us: gate + wire missing SSA SSI families#404
MaxGhenis merged 4 commits into
mainfrom
target-parity-contract

Conversation

@MaxGhenis

Copy link
Copy Markdown
Contributor

Target-parity contract for populace-us

The target-side analog of the input-column coverage contract (#368), now a launch gate. Every target the retired us-data/eCPS pipeline calibrated to — if it is a valid ledger-shaped administrative target — must be compiled into the populace registry, or carry a reviewed exclusion naming its evidence. Silent omission is the failure mode this kills. This gates Build M.

The gap this closes

The retired pipeline calibrated nation/ssa/ssi_recipients (SSA Annual Statistical Supplement — 7,404,820 SSI recipients in 2024). The pinned feed (consumer_facts_buildh_v8, sha 94b7155f) carries the ssa_supplement.ssi_recipients facts (208 rows: national + 51 states × aged/blind/disabled/total) and a parallel ssa_supplement.ssi_payments by-area table (208 rows). None of them compiled — the registry landed only the six oasdi_ssi_payments aggregates and dropped the recipient family with no record of the omission. Export-mass parity cannot catch this: a target family that never compiles contributes zero rows, not wrong ones.

Audit correction to the starting premise

The brief flagged the $63.079B / $51.075B SSI/OASDI payment measures and "core Social Security (OASDI) aggregates" as also uncompiled. The live compile on origin/main shows otherwise, and the contract records the true state:

  • ssi_payments = $63,079,493,000 compiles today as role ssi_total; social_security_dependents_benefits = $51,075,000,000 compiles as ssa_dependents_total. Both, plus all five OASDI aggregates, are the certified ssa_supplement.oasdi_ssi_payments:6 (added in Use Ledger-backed fiscal target references #65, present on main).
  • The genuinely-absent core SSA/SSI families are SSI recipients (national + state) and the state-level SSI payment table — plus the redundant national by-area SSI payment (a duplicate of ssi_total) and the aged/blind/disabled eligibility sub-rows.

So this PR wires the recipients and state payments; it does not re-wire OASDI/national-SSI payments (already present), and documents that finding rather than restating the premise.

What this PR does

1. Wires the missing SSA SSI families (fiscal_targets.py) — a source_name == "ssa" dispatch (_ssa_ssi_reference_from_fact) mirroring the census_pep/census_stc idiom:

  • oasdi_ssi_payments → delegated unchanged to _direct_reference_from_fact; the six certified ssa targets never regress.
  • ssi_recipients.by_area_category "total" → indicator sum of engine ssi receipt (role ssi_recipients), national + state (state_fips). Engine-computed counterpart, not a survey proxy — the same materialization path as the snap_households caseload (Compile USDA SNAP state and national caseload targets from FNS average-monthly facts #371), flipping measure_mode to indicator_sum on the same ssi variable the existing ssi_total dollar target already materializes.
  • ssi_payments.by_area_category "total" → dollar sum of engine ssi (role ssi_state_payments), state rows only.
  • aged/blind/disabled categories → dropped, documented as reviewed exclusions (PolicyEngine-US has no SSI eligibility-category split).

Registry 5,586 → 5,689 specs; national SSI recipients materialize to exactly 7,404,820; deterministic.

2. The contract (release_target_parity.py + target_parity_manifest.json) — a family-granularity register. A family is namespace.concept of the ledger source_record_id (us_target_family_id) — finer than the source family (so ssa alone can't hide the recipient gap behind compiled OASDI) and coarser than the row. Each family is compiled or reviewed_exclusion with a classification + reason + evidence.

3. The gate (us_release_target_parity_gate) — wired into build_us_fiscal_refresh_release.py on the compiled + substituted registry, before the diagnostic JCT skip. Fails the build when a compiled family is absent from the registry, or a reviewed exclusion has started compiling (cannot-rot, #286/#337).

4. Anti-rot (assert_target_parity_manifest_current) — the manifest declares exactly the feed's family surface (checked against the sha-pinned target_parity_feed_families.json, so it runs in CI without the 131 MB feed) plus documented source-absent us-data families, and the red line: the core ssa_supplement.ssi_recipients family must stay compiled and can never be quietly downgraded.

5. Tests — 6 SSA wiring cases + 28 contract cases mirroring the input-coverage suite (gate green/red/cannot-rot, anti-rot drift + red-line + feed reconciliation, guarded regeneration proving the committed artifacts reproduce from the pinned feed).

The classified diff — (us-data targets ∪ feed families) − registry

Compiled — 25 families (registry targets | feed facts):

family targets feed
cbo.revenue_projection 5 30
census_pep.national_resident_population_age 18 19
census_pep.v2024 918 969
census_stc.individual_income_tax_collections 44 46
cms_aca.oep2024 102 153
cms_medicaid.state_enrollment 135 475
cms_medicare.part_b_premium_income 1 1
hhs_acf_tanf.cash_assistance 30 52
irs_soi.filing_season_week47 34 34
irs_soi.historic_table_2 3888 5654
irs_soi.state_2022 2 4
irs_soi.table_1_1 / 1_2 / 1_4 / 2_1 / 2_5 / 4_3 2 / 4 / 22 / 22 / 235 / 9 84 / 7 / 557 / 39 / 472 / 18
jct.tax_expenditures 5 5
ssa_supplement.oasdi_ssi_payments 6 6
ssa_supplement.ssi_recipients (wired) 52 208
ssa_supplement.ssi_payments (wired) 51 208
usda_snap.national_average_monthly_households / national_benefits / state_average_monthly_households / state_benefits 1 / 1 / 51 / 51 1 / 1 / 53 / 53

Reviewed exclusions — 55 families, each with evidence:

classification families feed facts rationale (abridged)
macro_control_total 37 447 BEA NIPA national + regional aggregates, CBO total revenue, Fed Z.1 net worth — macro control totals with no household-linear counterpart; income fit from IRS SOI + CBO instead
survey_derived 1 468 census_acs.acs1_2023 — ACS is a sample survey; also CD-only, off by default
off_by_default 1 26,880 irs_soi.congressional_district_2022 — CD targets opt-in for the national build
superseded 2 138 census.popproj2023 (superseded by observed census_pep), kff.marketplace_effectuated_enrollment (superseded by CMS cms_aca)
not_modeled 3 111 SNAP persons (overcounts ~50% vs households — the #371 rationale), W-2 SS tips (no tip_income support)
deferred 3 60 TANF family/recipient caseloads (dollars compiled, counts not), LIHEAP national profile
input_side 4 6 401(k)/IRA contribution aggregates — governed as imputed input columns, not targets
non_linear 1 1 cms_nhe.medicaid_title_xix_expenditures — already validation_only (person-weight-dependent allocation)
source_absent 3 0 us-data targets with no feed fact: BLS Consumer Expenditure, WIC national summary, HUD housing assistance

Provenance

us-data target surface enumerated from the archived policyengine-us-data (db/etl_national_targets.py, db/etl_*.py, utils/national_target_parity.py), read via the GitHub archive, not the local fork. Feed inventory + registry compile are sha-pinned and reproduce deterministically (tools/build_us_target_parity_manifest.py).

Ruff clean; 243 tests green across the three touched suites. Please do not merge — this is for conductor review. Gates Build M.

🤖 Generated with Claude Code

MaxGhenis and others added 4 commits July 12, 2026 22:05
…rget registry

The retired us-data/eCPS pipeline calibrated SSI recipients
(nation/ssa/ssi_recipients) but the populace registry silently dropped them.
The v8 consumer feed carries ssa_supplement.ssi_recipients (7,404,820 national
plus 51 states) and ssa_supplement.ssi_payments by area/category, none of which
compiled — only the six oasdi_ssi_payments aggregates did.

Add a source_name == "ssa" dispatch (_ssa_ssi_reference_from_fact) mirroring the
census_pep / census_stc idiom:
- oasdi_ssi_payments: delegated unchanged to _direct_reference_from_fact, so the
  six certified ssa targets (social_security_total ... ssi_total) never regress.
- ssi_recipients.by_area_category "total": indicator sum of engine ssi receipt
  (role ssi_recipients), national and state (state_fips) grain.
- ssi_payments.by_area_category "total": dollar sum of engine ssi (role
  ssi_state_payments), state rows only — the national all_areas_total duplicates
  the OASDI-table ssi_total already compiled.
- aged/blind/disabled eligibility categories: dropped here, documented as
  reviewed exclusions (PolicyEngine-US has no SSI eligibility-category split).

Registry 5,586 -> 5,689 specs, deterministic. Engine-computed counterparts
(ssi), not reported-survey proxies. Six targeted tests.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The target-side analog of the input-column coverage contract, as a launch gate.
Every administrative target family the retired us-data/eCPS pipeline calibrated
to must be compiled into the populace registry or carry a reviewed exclusion
naming its evidence — silent omission is the failure this kills (the SSI
recipient family was dropped with no record of it).

- release_target_parity.py: family-id derivation (namespace.concept of the
  ledger source_record_id), the TargetFamily/TargetParityManifest model, the
  us_release_target_parity_gate (fails the build when a compiled family is
  absent from the registry, or a reviewed exclusion has started compiling —
  cannot-rot per #286/#337), and assert_target_parity_manifest_current
  (manifest vs the checked-in sha-pinned feed inventory, plus the red line that
  the core SSA SSI recipient family can never be downgraded off compiled).
- target_parity_manifest.json + target_parity_feed_families.json: generated,
  sha-pinned to consumer_facts_buildh_v8 (94b7155f). 25 compiled families, 55
  reviewed exclusions (52 feed-only, each with a classification + evidence, and
  3 source-absent us-data families: BLS CE, WIC, HUD housing).
- build_us_target_parity_manifest.py: regenerates both artifacts; halts if a
  feed family neither compiles nor has a declared, evidence-naming exclusion.
- Wired into build_us_fiscal_refresh_release.py on the compiled + substituted
  registry, before the diagnostic JCT skip, mirroring the input-coverage gate.
- 28 tests mirroring the input-coverage suite (gate green/red/cannot-rot,
  anti-rot drift + red-line, feed reconciliation, guarded regeneration).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
test_main_writes_diagnostics_before_post_calibration_gate_failure drives main()
with a stub target registry, so it must stub the new target-parity gate and
anti-rot exactly as it already stubs target_profile_coverage_gate and
assert_release_input_coverage_manifest_current (a stub registry cannot carry the
full 25-family surface the real gate requires).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…bent-repo token

CI (wheels + spec-only tests) surfaced two shipping-contract violations:
- the us country package must declare every file it ships: add
  target_parity_manifest.json and target_parity_feed_families.json to
  country_package.json resources.
- the live tree may not name the retired incumbent data package: reword the
  source-absent provenance / us_data_source strings in the manifest generator
  (and regenerate the manifest) to "retired us-data pipeline (archived)"
  instead of the hyphenated incumbent package name, which
  test_no_incumbent_data_package_references_in_live_tree forbids outside the
  three allowlisted eCPS-parity-reference files.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant