Skip to content

feat(samples): add reviewed extraction proof packages - #10

Merged
jdrhyne merged 2 commits into
mainfrom
codex/plg-reviewed-examples-release
Aug 20, 2026
Merged

feat(samples): add reviewed extraction proof packages#10
jdrhyne merged 2 commits into
mainfrom
codex/plg-reviewed-examples-release

Conversation

@jdrhyne

@jdrhyne jdrhyne commented Aug 19, 2026

Copy link
Copy Markdown
Collaborator

Summary

  • Add three runnable, reviewed Data Extraction examples for mortgage verification, insurance claim intake, and prior authorization using authentic public forms filled with privacy-safe demo values.
  • Publish a GitHub Pages evidence hub that shows exact-value comparisons, returned primary source regions, and retained review issues without presenting the selected forms as an accuracy or performance benchmark.
  • Add deterministic fixture builders, receipt-bound replay/checking, release screenshots, privacy/claim boundaries, and regression coverage for the proof packages.

Selected-form results

  • Mortgage verification: 6/8 exact, 8 primary regions returned, 2 retained issues.
  • Insurance claim intake: 11/11 exact, 11 primary regions returned, 0 issues; public route wording is 11/11 exact → Continue while the technical checker retains Pass.
  • Prior authorization: 9/11 exact, 11 primary regions returned, 2 retained issues.

These are reviewed results for the committed forms, not a benchmark or production-accuracy claim.

Test plan

  • python3 -m pytest -q -p no:cacheprovider — 271 passed, 1 skipped.
  • Offline replay and checker exit contracts: mortgage 1/1, insurance 0/0, prior authorization 1/1.
  • Five responsive hub captures plus desktop/mobile proof captures reviewed for clipping and claim consistency.
  • GitHub Pages proof copies verified byte-identical to the reviewed outputs.
  • Full-history secret scan completed with gitleaks v8.30.1 and no findings.
  • Release claim, privacy, source-hash, cache/receipt, link-ledger, and deterministic-builder checks passed.

Rollout

After merge, verify the Pages hub and all three proof URLs return HTTP 200 before merging the dependent nutrient-website PR. Social/community publication is intentionally separate.

Add three authentic public-form examples with receipt-bound evidence, offline replays, and a Pages-ready proof hub. Preserve retained mismatches and enforce the claim, privacy, and returned-region boundaries in tests.
@jdrhyne jdrhyne added the enhancement New feature or request label Aug 19, 2026
@jdrhyne jdrhyne self-assigned this Aug 19, 2026
@jdrhyne
jdrhyne requested review from CosminIorgus and marijatrpkovic and a balanced review from Copilot August 19, 2026 21:29

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds reviewed extraction proof packages for mortgage, insurance, and prior-authorization forms, plus reproducible evidence, deterministic fixture tooling, and regression checks.

Changes:

  • Adds three authentic-form demo packages with reviewed live-response evidence.
  • Publishes proof documentation, design assets, and reproducibility tooling.
  • Expands cache, rendering, convergence, and evidence-contract tests.

Reviewed changes

Copilot reviewed 64 out of 115 changed files in this pull request and generated 4 comments.

Show a summary per file
File Description
tests/test_transports.py Updates the Parse API route assertion.
tests/test_reproducible_proof.py Tests reproducible proof artifacts and checker behavior.
tests/test_render.py Tests rendering AcroForm values.
tests/test_lighthouse_packages.py Validates hub claims, links, assets, and proofs.
tests/test_convergence.py Pins regenerated and reviewed artifact hashes.
tests/test_cache.py Adds reviewed current/legacy cache inventory checks.
scripts/build_lighthouse_fixtures.py Builds or verifies synthetic fixtures.
README.md Documents eight demos and reviewed results.
docs/reproducible-proof.md Provides offline replay and live-request instructions.
docs/lighthouse/assets/nutrient-logo.svg Adds the deployed logo asset.
docs/lighthouse-design-source.md Records the frozen design source and claims.
demos/seed_manifest.json Registers reviewed and legacy evidence.
demos/prior_authorization/README.md Documents the prior-authorization package.
demos/prior_authorization/provisional/evidence.json Adds provisional evidence.
demos/prior_authorization/output/comparison.json Records reviewed comparison results.
demos/prior_authorization/generate_demo.py Configures package generation.
demos/prior_authorization/fixture.json Defines fixture provenance and fields.
demos/prior_authorization/expected.json Defines the source oracle.
demos/prior_authorization/docs.json Defines the extraction schema.
demos/prior_authorization/check_expected.py Adds the package checker.
demos/prior_authorization/cache/8b71…receipt.json Adds the legacy receipt.
demos/prior_authorization/cache/8b71….json Preserves the legacy response.
demos/prior_authorization/cache/18bb…receipt.json Adds the reviewed receipt.
demos/prior_authorization/cache/18bb….json Adds the reviewed response.
demos/prior_authorization/build_public_form.py Builds the CMS-derived demo PDF.
demos/mortgage_verification/source/README.md Records source provenance.
demos/mortgage_verification/README.md Documents the mortgage package.
demos/mortgage_verification/provisional/evidence.json Adds provisional evidence.
demos/mortgage_verification/output/comparison.json Records reviewed comparison results.
demos/mortgage_verification/generate_demo.py Configures package generation.
demos/mortgage_verification/fixture.json Defines fixture provenance and fields.
demos/mortgage_verification/expected.json Defines the source oracle.
demos/mortgage_verification/docs.json Defines the extraction schema.
demos/mortgage_verification/check_expected.py Adds the package checker.
demos/mortgage_verification/cache/2eab…receipt.json Adds the reviewed receipt.
demos/mortgage_verification/cache/2eab….json Adds the reviewed response.
demos/mortgage_verification/cache/15f9…receipt.json Adds the legacy receipt.
demos/mortgage_verification/cache/15f9….json Preserves the legacy response.
demos/mortgage_verification/build_public_form.py Builds the Treasury-derived demo PDF.
demos/insurance_claim_intake/README.md Documents the insurance package.
demos/insurance_claim_intake/provisional/evidence.json Adds provisional evidence.
demos/insurance_claim_intake/output/comparison.json Records the passing reviewed result.
demos/insurance_claim_intake/generate_demo.py Configures package generation.
demos/insurance_claim_intake/fixture.json Defines fixture provenance and fields.
demos/insurance_claim_intake/expected.json Defines the source oracle.
demos/insurance_claim_intake/docs.json Defines the extraction schema.
demos/insurance_claim_intake/check_expected.py Adds the package checker.
demos/insurance_claim_intake/cache/acb9…receipt.json Adds the legacy receipt.
demos/insurance_claim_intake/cache/acb9….json Preserves the legacy response.
demos/insurance_claim_intake/cache/82bd…receipt.json Adds the reviewed receipt.
demos/insurance_claim_intake/cache/82bd….json Adds the reviewed response.
demos/insurance_claim_intake/build_fixture.py Builds and verifies the GSA fixture.
demos/grounded_extraction/instructions.json Publishes extraction instructions.
demos/grounded_extraction/expected.json Adds reviewed source truth.
demos/grounded_extraction/check_expected.py Adds deterministic result checking.
common/render.py Enables AcroForm rendering.
common/assets/logotype/nutrient-logo.svg Adds the shared logo asset.
.gitignore Allows the fixture-building script.
.gitattributes Marks PDFs as binary.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread demos/grounded_extraction/check_expected.py Outdated
Comment thread tests/test_cache.py Outdated
Comment thread demos/grounded_extraction/instructions.json
Comment thread demos/prior_authorization/build_public_form.py Outdated
@sashamilenkovic
sashamilenkovic self-requested a review August 19, 2026 23:04

@sashamilenkovic sashamilenkovic left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGT?M

Things to consider

  1. Undeclared builder dependenciesbuild_public_form.py (mortgage, prior-auth) imports PIL, pypdf, and reportlab, none of which are in requirements*.txt, so the README-documented rebuild commands fail on a clean checkout. The builders also assert exact SHA-256s through these libraries' output, so the versions should be pinned for the reproducibility claim to hold across machines.

  2. The documented offline quick-start overwrites pinned artifactsgenerate_demo.py --provisional and check_expected.py --allow-provisional rewrite output/comparison.json and index.html, which are SHA-pinned in seed_manifest.json and the tests. Following the README then running pytest produces failures with no hint that --replay-reviewed restores the bytes. Consider writing provisional output elsewhere, or documenting the restore path.

  3. Prior-auth builder can silently replace the reviewed inputbuild() writes data/prior-authorization.pdf before comparing against fixture.json's pinned sourceSha256 and exits 0; no --check mode or pinned digest, unlike both sibling builders. Compare first, then write.

  4. Font licensing — commercial ABC Monument Grotesk woff2 files are committed twice in a public MIT-licensed repo and base64-embedded in the published pages. Worth confirming redistribution is covered.

  5. test_cache.py weakened the exact cache inventory to a subset check — a stray unaccounted demos/*/cache/*.json now passes. Set equality holds today (17 = 5 seeded + 12 reviewed); restoring it is a small fix.

  6. grounded_extraction/check_expected.py::_has_grounding is lax — accepts any pageIndex ≥ 0, negative origins, and out-of-page boxes. Not exploitable today (the checker intentionally exits 1), but reproducible-proof.md leans on this script; consider aligning it with the strict lighthouse validation.

  7. Schema/oracle gap in grounded_extractionprior_authorization_number is in expected.json but not the schema's required list, so a future refresh could legally omit it (fails closed). Fixing changes the cache key and needs a new live call + re-review — reasonable to defer to the next refresh.

  8. Internal details in docs/lighthouse-design-source.md — local machine path, "agent-memory," DNS anecdote. The commit pin and asset SHA table carry the provenance.

  9. Smaller notesseed_cache.py re-implements receipt/digest validation that lighthouse.py owns; a reviewed-response refresh requires hash updates in three files (worth a pointer comment); the mortgage builder requires the sibling rma_extraction PDF even when its frozen copy exists; _validate_json_tree mislabels shared-but-acyclic references as "cyclic"; checkbox oracle values are semantic while the docstring says "visibly printed"; design-source doc says insurance is "marked Pass" vs. the hub's "11/11 exact → Continue".

@jdrhyne

jdrhyne commented Aug 20, 2026

Copy link
Copy Markdown
Collaborator Author

Review follow-up is pushed in 4cf4f77:

  • pinned the missing Pillow, pypdf, and ReportLab builder dependencies; a fresh temporary venv from requirements*.txt passes the full suite (277 passed, 1 skipped)
  • moved provisional render artifacts under provisional/output/ and added regressions proving provisional generation/checking cannot overwrite reviewed output
  • made the prior-authorization builder validate in memory, verify the pinned digest before an atomic write, and support non-writing --check
  • removed the redistributed ABC Monument font binaries and embedded font data; regenerated the reviewed pages and screenshots with system stacks
  • restored exact cache-inventory equality
  • made the grounded-extraction checker validate page references and full page-bounded boxes, with regressions
  • documented the prior_authorization_number required-field change as a next-live-refresh gate because it changes the request digest/cache key and requires a new authorized call plus independent review; no live call was made here
  • removed local path/agent-memory/DNS details from the public design-source document and reconciled its insurance wording

Offline reviewed checks still retain the intended evidence envelope: mortgage 6/8 with two issues, insurance 11/11 pass, and prior authorization 9/11 with two issues. Desktop/mobile browser QA of the hub and all three proofs found no overflow, broken images, console errors, or warnings.

The lower-priority structural cleanups in note 9 (for example consolidating receipt validation) are intentionally left out of this review-fix commit because they are not needed to close the release correctness and redistribution concerns.

@jdrhyne
jdrhyne merged commit 17edb73 into main Aug 20, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants