Skip to content

Document YAML unit-test format and portability notes#517

Open
edithatogo wants to merge 2 commits into
PolicyEngine:masterfrom
edithatogo:docs/yaml-test-portability
Open

Document YAML unit-test format and portability notes#517
edithatogo wants to merge 2 commits into
PolicyEngine:masterfrom
edithatogo:docs/yaml-test-portability

Conversation

@edithatogo

@edithatogo edithatogo commented Jul 9, 2026

Copy link
Copy Markdown

What's changed

Closes #514.

Documents the YAML unit-test shape accepted by policyengine_core.tools.test_runner, including dotted-input parameter reform behavior and portability guidance for deterministic external converters. Native PolicyEngine YAML remains the in-repository source of truth; no converter or runtime dependency is added to core.

Verification

  • Documentation page added to the table of contents.
  • Changelog fragment added.
  • Examples and rejection guidance are grounded in the current loader behavior.
  • make format && make documentation awaits the upstream workflow run.
  • Issue linked and project item updated.

Compatibility and limits

Documentation-only. The external prototype supports a deliberately restricted subset and rejects reforms, extensions, relative margins, lists, nulls, and expressions that cannot be translated without loss.

Maintainer action required

Please approve the first-time-contributor documentation workflow and confirm whether this format guidance belongs in core documentation.

@MaxGhenis MaxGhenis left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for this — keeping the converter prototype out of core and documenting the native format instead is the right call, and the page fills a real gap. I reviewed the doc against test_runner.py.

Accuracy check (verified against policyengine_core/tools/test_runner.py)

  • Accepted top-level fields match TEST_KEYWORDS exactly (absolute_error_margin, description, extensions, ignore_variables, input, keywords, name, only_variables, output, period, reforms, relative_error_margin). ✔
  • "Unknown top-level keys are rejected" matches the TEST_KEYWORDS.issuperset(test.keys()) check that raises Unexpected keys …. ✔
  • output required at runtime matches the missing-output guard. ✔
  • Dotted input keys → inline parameter reform via set_parameter is consistent with the code (set_parameter is imported and inline_reforms is assembled during input parsing). ✔

So the documentation is factually accurate against the runner, which is the main thing that matters for a docs page. Changelog fragment is present, and the headings are already sentence case per our style.

Suggestions (minor)

  1. Confirm the docs build passes in CI. The new docs/usage/yaml_tests.md and the _toc.yml entry look right; just make sure make documentation renders the page and the Markdown table under MyST — I didn't run the full docs build locally, and the CI Test job does build docs.
  2. Link the mechanism. Where the doc notes dotted keys "differ from OpenFisca Core," a pointer to test_runner.py / the set_parameter path would help a reader find where that behavior lives.
  3. Optional synergy with #515. If the policyengine.trace.v1 export in #515 lands, a short companion schema page in this same usage section would give that format a documented home. Not needed for this PR.

Nice, tightly-scoped contribution.

@codecov

codecov Bot commented Jul 9, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 86.26%. Comparing base (c94a745) to head (455006c).
⚠️ Report is 37 commits behind head on master.

Additional details and impacted files
@@            Coverage Diff             @@
##           master     #517      +/-   ##
==========================================
+ Coverage   84.69%   86.26%   +1.56%     
==========================================
  Files         223      230       +7     
  Lines       11830    12774     +944     
  Branches     1169     1240      +71     
==========================================
+ Hits        10019    11019    +1000     
+ Misses       1496     1416      -80     
- Partials      315      339      +24     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Add a usage page describing the test_runner YAML shape, dotted-key
parameter reform behavior, and guidance for external portable converters.
Keeps converters out of core while answering issue PolicyEngine#514.

Closes PolicyEngine#514
Link the OpenFisca-differing parameter override path to the runner
implementation Max asked for in review.

Co-authored-by: Cursor <cursoragent@cursor.com>
@edithatogo

Copy link
Copy Markdown
Author

Follow-up to @MaxGhenis review

Added an explicit pointer from the dotted-key / set_parameter note to policyengine_core/tools/test_runner.py (where inline_reforms is assembled).

Could you approve the workflow runs so the docs build in CI can confirm the MyST page? Optional companion schema page for policyengine.trace.v1 can follow #515 if you want it in-tree.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Proposal: deterministic PolicyEngine YAML test -> portable fixture converter

2 participants