feat(samples): add reviewed extraction proof packages - #10
Conversation
Add three authentic public-form examples with receipt-bound evidence, offline replays, and a Pages-ready proof hub. Preserve retained mismatches and enforce the claim, privacy, and returned-region boundaries in tests.
There was a problem hiding this comment.
Pull request overview
Adds reviewed extraction proof packages for mortgage, insurance, and prior-authorization forms, plus reproducible evidence, deterministic fixture tooling, and regression checks.
Changes:
- Adds three authentic-form demo packages with reviewed live-response evidence.
- Publishes proof documentation, design assets, and reproducibility tooling.
- Expands cache, rendering, convergence, and evidence-contract tests.
Reviewed changes
Copilot reviewed 64 out of 115 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
tests/test_transports.py |
Updates the Parse API route assertion. |
tests/test_reproducible_proof.py |
Tests reproducible proof artifacts and checker behavior. |
tests/test_render.py |
Tests rendering AcroForm values. |
tests/test_lighthouse_packages.py |
Validates hub claims, links, assets, and proofs. |
tests/test_convergence.py |
Pins regenerated and reviewed artifact hashes. |
tests/test_cache.py |
Adds reviewed current/legacy cache inventory checks. |
scripts/build_lighthouse_fixtures.py |
Builds or verifies synthetic fixtures. |
README.md |
Documents eight demos and reviewed results. |
docs/reproducible-proof.md |
Provides offline replay and live-request instructions. |
docs/lighthouse/assets/nutrient-logo.svg |
Adds the deployed logo asset. |
docs/lighthouse-design-source.md |
Records the frozen design source and claims. |
demos/seed_manifest.json |
Registers reviewed and legacy evidence. |
demos/prior_authorization/README.md |
Documents the prior-authorization package. |
demos/prior_authorization/provisional/evidence.json |
Adds provisional evidence. |
demos/prior_authorization/output/comparison.json |
Records reviewed comparison results. |
demos/prior_authorization/generate_demo.py |
Configures package generation. |
demos/prior_authorization/fixture.json |
Defines fixture provenance and fields. |
demos/prior_authorization/expected.json |
Defines the source oracle. |
demos/prior_authorization/docs.json |
Defines the extraction schema. |
demos/prior_authorization/check_expected.py |
Adds the package checker. |
demos/prior_authorization/cache/8b71…receipt.json |
Adds the legacy receipt. |
demos/prior_authorization/cache/8b71….json |
Preserves the legacy response. |
demos/prior_authorization/cache/18bb…receipt.json |
Adds the reviewed receipt. |
demos/prior_authorization/cache/18bb….json |
Adds the reviewed response. |
demos/prior_authorization/build_public_form.py |
Builds the CMS-derived demo PDF. |
demos/mortgage_verification/source/README.md |
Records source provenance. |
demos/mortgage_verification/README.md |
Documents the mortgage package. |
demos/mortgage_verification/provisional/evidence.json |
Adds provisional evidence. |
demos/mortgage_verification/output/comparison.json |
Records reviewed comparison results. |
demos/mortgage_verification/generate_demo.py |
Configures package generation. |
demos/mortgage_verification/fixture.json |
Defines fixture provenance and fields. |
demos/mortgage_verification/expected.json |
Defines the source oracle. |
demos/mortgage_verification/docs.json |
Defines the extraction schema. |
demos/mortgage_verification/check_expected.py |
Adds the package checker. |
demos/mortgage_verification/cache/2eab…receipt.json |
Adds the reviewed receipt. |
demos/mortgage_verification/cache/2eab….json |
Adds the reviewed response. |
demos/mortgage_verification/cache/15f9…receipt.json |
Adds the legacy receipt. |
demos/mortgage_verification/cache/15f9….json |
Preserves the legacy response. |
demos/mortgage_verification/build_public_form.py |
Builds the Treasury-derived demo PDF. |
demos/insurance_claim_intake/README.md |
Documents the insurance package. |
demos/insurance_claim_intake/provisional/evidence.json |
Adds provisional evidence. |
demos/insurance_claim_intake/output/comparison.json |
Records the passing reviewed result. |
demos/insurance_claim_intake/generate_demo.py |
Configures package generation. |
demos/insurance_claim_intake/fixture.json |
Defines fixture provenance and fields. |
demos/insurance_claim_intake/expected.json |
Defines the source oracle. |
demos/insurance_claim_intake/docs.json |
Defines the extraction schema. |
demos/insurance_claim_intake/check_expected.py |
Adds the package checker. |
demos/insurance_claim_intake/cache/acb9…receipt.json |
Adds the legacy receipt. |
demos/insurance_claim_intake/cache/acb9….json |
Preserves the legacy response. |
demos/insurance_claim_intake/cache/82bd…receipt.json |
Adds the reviewed receipt. |
demos/insurance_claim_intake/cache/82bd….json |
Adds the reviewed response. |
demos/insurance_claim_intake/build_fixture.py |
Builds and verifies the GSA fixture. |
demos/grounded_extraction/instructions.json |
Publishes extraction instructions. |
demos/grounded_extraction/expected.json |
Adds reviewed source truth. |
demos/grounded_extraction/check_expected.py |
Adds deterministic result checking. |
common/render.py |
Enables AcroForm rendering. |
common/assets/logotype/nutrient-logo.svg |
Adds the shared logo asset. |
.gitignore |
Allows the fixture-building script. |
.gitattributes |
Marks PDFs as binary. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
sashamilenkovic
left a comment
There was a problem hiding this comment.
LGT?M
Things to consider
-
Undeclared builder dependencies —
build_public_form.py(mortgage, prior-auth) importsPIL,pypdf, andreportlab, none of which are inrequirements*.txt, so the README-documented rebuild commands fail on a clean checkout. The builders also assert exact SHA-256s through these libraries' output, so the versions should be pinned for the reproducibility claim to hold across machines. -
The documented offline quick-start overwrites pinned artifacts —
generate_demo.py --provisionalandcheck_expected.py --allow-provisionalrewriteoutput/comparison.jsonandindex.html, which are SHA-pinned inseed_manifest.jsonand the tests. Following the README then runningpytestproduces failures with no hint that--replay-reviewedrestores the bytes. Consider writing provisional output elsewhere, or documenting the restore path. -
Prior-auth builder can silently replace the reviewed input —
build()writesdata/prior-authorization.pdfbefore comparing againstfixture.json's pinnedsourceSha256and exits 0; no--checkmode or pinned digest, unlike both sibling builders. Compare first, then write. -
Font licensing — commercial ABC Monument Grotesk woff2 files are committed twice in a public MIT-licensed repo and base64-embedded in the published pages. Worth confirming redistribution is covered.
-
test_cache.pyweakened the exact cache inventory to a subset check — a stray unaccounteddemos/*/cache/*.jsonnow passes. Set equality holds today (17 = 5 seeded + 12 reviewed); restoring it is a small fix. -
grounded_extraction/check_expected.py::_has_groundingis lax — accepts any pageIndex ≥ 0, negative origins, and out-of-page boxes. Not exploitable today (the checker intentionally exits 1), butreproducible-proof.mdleans on this script; consider aligning it with the strict lighthouse validation. -
Schema/oracle gap in grounded_extraction —
prior_authorization_numberis inexpected.jsonbut not the schema'srequiredlist, so a future refresh could legally omit it (fails closed). Fixing changes the cache key and needs a new live call + re-review — reasonable to defer to the next refresh. -
Internal details in
docs/lighthouse-design-source.md— local machine path, "agent-memory," DNS anecdote. The commit pin and asset SHA table carry the provenance. -
Smaller notes —
seed_cache.pyre-implements receipt/digest validation thatlighthouse.pyowns; a reviewed-response refresh requires hash updates in three files (worth a pointer comment); the mortgage builder requires the siblingrma_extractionPDF even when its frozen copy exists;_validate_json_treemislabels shared-but-acyclic references as "cyclic"; checkbox oracle values are semantic while the docstring says "visibly printed"; design-source doc says insurance is "markedPass" vs. the hub's "11/11 exact → Continue".
|
Review follow-up is pushed in
Offline reviewed checks still retain the intended evidence envelope: mortgage The lower-priority structural cleanups in note 9 (for example consolidating receipt validation) are intentionally left out of this review-fix commit because they are not needed to close the release correctness and redistribution concerns. |
Summary
Selected-form results
11/11 exact → Continuewhile the technical checker retainsPass.These are reviewed results for the committed forms, not a benchmark or production-accuracy claim.
Test plan
python3 -m pytest -q -p no:cacheprovider— 271 passed, 1 skipped.Rollout
After merge, verify the Pages hub and all three proof URLs return HTTP 200 before merging the dependent nutrient-website PR. Social/community publication is intentionally separate.