Document YAML unit-test format and portability notes#517
Conversation
MaxGhenis
left a comment
There was a problem hiding this comment.
Thanks for this — keeping the converter prototype out of core and documenting the native format instead is the right call, and the page fills a real gap. I reviewed the doc against test_runner.py.
Accuracy check (verified against policyengine_core/tools/test_runner.py)
- Accepted top-level fields match
TEST_KEYWORDSexactly (absolute_error_margin, description, extensions, ignore_variables, input, keywords, name, only_variables, output, period, reforms, relative_error_margin). ✔ - "Unknown top-level keys are rejected" matches the
TEST_KEYWORDS.issuperset(test.keys())check that raisesUnexpected keys …. ✔ outputrequired at runtime matches the missing-outputguard. ✔- Dotted input keys → inline parameter reform via
set_parameteris consistent with the code (set_parameteris imported andinline_reformsis assembled during input parsing). ✔
So the documentation is factually accurate against the runner, which is the main thing that matters for a docs page. Changelog fragment is present, and the headings are already sentence case per our style.
Suggestions (minor)
- Confirm the docs build passes in CI. The new
docs/usage/yaml_tests.mdand the_toc.ymlentry look right; just make suremake documentationrenders the page and the Markdown table under MyST — I didn't run the full docs build locally, and the CITestjob does build docs. - Link the mechanism. Where the doc notes dotted keys "differ from OpenFisca Core," a pointer to
test_runner.py/ theset_parameterpath would help a reader find where that behavior lives. - Optional synergy with #515. If the
policyengine.trace.v1export in #515 lands, a short companion schema page in this same usage section would give that format a documented home. Not needed for this PR.
Nice, tightly-scoped contribution.
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #517 +/- ##
==========================================
+ Coverage 84.69% 86.26% +1.56%
==========================================
Files 223 230 +7
Lines 11830 12774 +944
Branches 1169 1240 +71
==========================================
+ Hits 10019 11019 +1000
+ Misses 1496 1416 -80
- Partials 315 339 +24 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
Add a usage page describing the test_runner YAML shape, dotted-key parameter reform behavior, and guidance for external portable converters. Keeps converters out of core while answering issue PolicyEngine#514. Closes PolicyEngine#514
455006c to
738c8d8
Compare
Link the OpenFisca-differing parameter override path to the runner implementation Max asked for in review. Co-authored-by: Cursor <cursoragent@cursor.com>
Follow-up to @MaxGhenis reviewAdded an explicit pointer from the dotted-key / Could you approve the workflow runs so the docs build in CI can confirm the MyST page? Optional companion schema page for |
What's changed
Closes #514.
Documents the YAML unit-test shape accepted by
policyengine_core.tools.test_runner, including dotted-input parameter reform behavior and portability guidance for deterministic external converters. Native PolicyEngine YAML remains the in-repository source of truth; no converter or runtime dependency is added to core.Verification
make format && make documentationawaits the upstream workflow run.Compatibility and limits
Documentation-only. The external prototype supports a deliberately restricted subset and rejects reforms, extensions, relative margins, lists, nulls, and expressions that cannot be translated without loss.
Maintainer action required
Please approve the first-time-contributor documentation workflow and confirm whether this format guidance belongs in core documentation.