Promotion certification suite: pre-registered out-of-sample scorecard at default-flip time

[MaxGhenis]

Promotion certification suite: pre-registered out-of-sample scorecard at default-flip time

Problem

Datasets become the policyengine.py default via a pin-bump PR whose evidence is currently assembled ad hoc (e.g. #427's matched-N comparison for the UK, per-build gate reports for the US). Three gaps:

1. Authority location. Backtests computed inside populace at build time are diagnostics (right — see populace#348); nothing with pass/fail authority runs at the promotion boundary, where the question is "is this file fit to be the default under the engine users will actually run".
2. Engine entanglement. Build-time scorecards are stamped under the build's engine pin; policyengine.py ships newer engines, so those scorecards go stale. Reform deltas vs external scores test dataset × engine jointly and only make sense evaluated with the shipping engine.
3. Goodhart containment. Without pre-registration and a ledger of evaluated candidates, iterating builds against held-out metrics quietly converts them into targets.

Proposal

A certification runner in policyengine.py that the default-flip PR must carry:

• Inputs: candidate dataset URI (sha-pinned), the engine version the bundle will resolve, a pre-registered metric set + bands (frozen in-repo before the candidate exists; changes to the set are PRs with their own review).
• Output: one scorecard artifact attached to the pin-bump PR, plus a ledger line recording every candidate evaluated (so "N builds were tried before one passed" is visible).
• Metric families (per country): population-level backtests never used as calibration targets — poverty rates vs official statistics, program caseloads/expenditures vs administrative totals, reform deltas vs external scores (JCT/CBO; OBR/HMRC) — plus mechanical fitness checks (column-manifest completeness per populace#340, weight-health summary).
• Relationship to populace: populace keeps build-integrity gates + freely consulted diagnostics (populace#348, UK: structural reforms gated on *.in_effect=True silently no-op via simulation_modifier path #302); the promotion suite is evaluated once per candidate at the boundary. Fix path for a failed metric is inputs/joint structure in populace, never fitting the output.

First two instances

US — next default flip (populace#299 Build H candidate). Scorecard: SPM poverty (national + state, Census P60-287, the populace#348 rows), SNAP/SSI/Medicaid caseload-and-cost vs administrative totals, OBBBA reform deltas vs JCT scores (with the #340 absent-column families explicitly reported), income tax / Social Security aggregates vs SOI/SSA under the shipping engine.

UK — retroactive adjudication of populace-uk vs enhanced FRS. The #427 promotion rested on a sound matched-N holdout comparison (6/6 rotations) — strong but scoped to the calibration surface, and per-target it was 79–70. The suite upgrade: popdgp joint-distribution metrics (energy, C2ST) plus out-of-sample backtests — HBAI poverty rates (overall/child/pensioner, BHC/AHC), DWP benefit caseloads and expenditure (UC, state pension, PIP), HMRC income-tax liabilities by band — run identically over both artifacts under policyengine-uk at the bundle pin. This either upgrades "wins the measured surface" to an unqualified claim we can publish, or finds where the incumbent still wins (e.g. the private-school-students-style targets). The transfer-paper eval-pack machinery (pre-registered config, frozen shas, scorecard-only output) is the template and makes this cheap.

Non-goals

• Not a release gate inside populace builds (selection-pressure containment; populace#302 covers holdout-masked build gates separately).
• Not continuous re-certification on every engine bump initially — start with flip-time evaluation; scheduled re-runs can come later.

Refs: populace#348 (poverty backtests as diagnostics), populace#302 (holdout-masked build gates), populace#305 (validation-portfolio META), populace#340 (column manifest), #427 (UK promotion evidence), populace#8 (UK sound comparison).

[MaxGhenis]

Engine policyengine-uk==2.89.2, period 2023; popdgp 0.1.0 (402fb23); candidate populace_uk n=535,080 vs incumbent enhanced_frs n=53,508 households.

Part A — popdgp joint-distribution (descriptive divergence)

Symmetric divergence between the two artifacts' derived-outcome distributions; energy shown as a ratio to each artifact's own sampling-noise floor. Does not pick a winner.

view	C2ST AUC (pop vs FRS)	verdict	energy	×incumbent floor	×populace floor
income	0.609	intermediate	0.0304	2.8×	16.1×
tax_benefit	0.675	intermediate	0.0582	3.7×	34.2×
net_income	0.656	intermediate	0.0384	3.4×	40.6×
full	0.720	intermediate	0.0622	3.0×	23.0×

popdgp (descriptive): 0/4 views clearly distinct, 0/4 indistinguishable between populace-uk and enhanced FRS.

Part B — out-of-sample adjudication (lower |error| wins)

metric	benchmark	populace-uk	err	enhanced FRS	err	winner
poverty_all_bhc	17.2%	19.3%	+2.1pp	13.4%	-3.8pp	populace
poverty_all_ahc	21.1%	23.9%	+2.8pp	16.9%	-4.2pp	populace
poverty_child_bhc	23.2%	25.1%	+1.9pp	18.5%	-4.7pp	populace
poverty_child_ahc	30.5%	34.2%	+3.7pp	26.0%	-4.5pp	populace
poverty_pensioner_bhc	18.6%	19.6%	+1.0pp	16.4%	-2.2pp	populace
poverty_pensioner_ahc	15.7%	18.1%	+2.4pp	13.8%	-1.9pp	FRS
income_taxpayers_total	36.60m	34.52m	-5.7%	38.57m	+5.4%	tie
income_taxpayers_basic	29.61m	27.17m	-8.2%	34.43m	+16.3%	populace
income_taxpayers_higher	6.03m	6.10m	+1.1%	3.55m	-41.2%	populace
income_taxpayers_additional	0.92m	1.25m	+34.9%	0.60m	-35.4%	tie

Context — in-sample metrics (both calibrated to these; not adjudicated)

metric	benchmark	populace-uk	err	enhanced FRS	err
caseload_universal_credit	4.75m	3.27m	-31.1%	6.30m	+32.6%
caseload_state_pension	12.75m	12.43m	-2.6%	11.61m	-9.0%
caseload_pip	3.15m	2.76m	-12.5%	3.62m	+15.0%
expenditure_universal_credit	£52.1bn	£34.7bn	-33.5%	£71.2bn	+36.6%
expenditure_state_pension	£124.1bn	£118.0bn	-4.9%	£103.9bn	-16.3%
expenditure_pip	£21.7bn	£17.9bn	-17.4%	£24.9bn	+14.7%
income_tax_total_liability	£277.0bn	£334.6bn	+20.8%	£247.6bn	-10.6%

Verdict

• Out-of-sample head-to-head: populace 6 / incumbent 1 / tie 1.
• Decisive incumbent wins: none.
• UK-data gaps (both miss badly, xref populace#145): ['poverty_child_ahc', 'caseload_universal_credit', 'expenditure_universal_credit', 'income_taxpayers_additional'].
• Unqualified superiority supported: False.

Out-of-sample head-to-head (OOS metrics only): populace 6 / incumbent 1 / tie 1. Unqualified superiority NOT supported; incumbent wins: none decisive but non-zero incumbent wins UK-data gaps (both miss badly): poverty_child_ahc, caseload_universal_credit, expenditure_universal_credit, income_taxpayers_additional.