Accept local dataset paths in managed_microsimulation#416
Conversation
resolve_managed_dataset_reference now treats a local filesystem path as an unmanaged dataset when allow_unmanaged=True -- the same explicit opt-in already required for remote URIs. materialize_dataset_source already passes non-URI paths through unchanged, so the simulation constructs normally and records the provenance bundle (managed_by=policyengine.py). Local build-and-score pipelines can run their own Stage-output H5s through the managed wrapper instead of constructing policyengine_us simulations directly. A local path without allow_unmanaged=True now raises an actionable error rather than the generic "Unknown dataset" message. Fixes #415 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
Closing as superseded (2026-07-04 worktree/PR sweep). Evidence: PR #442 (commit c1e33f0, "Use Populace geoslices in policyengine.py", merged 2026-06-28) already added the identical local_path/allow_unmanaged opt-in branch to resolve_managed_dataset_reference in src/policyengine/provenance/manifest.py, plus a matching fix to resolve_dataset_reference that #416 never touched. Main's tests/test_release_manifests.py already contains a test with the exact same name (test__given_local_path__then_managed_resolution_requires_opt_in, line 321) asserting the same opt-in-gated behavior, and the "unknown dataset name" case #416's third test checks is already covered by the pre-existing Unknown dataset error in resolve_dataset_reference. The head branch and a rescue of its worktree state are preserved locally and in git bundles under ~/.claude/backups/worktree-sweep-20260704/. Reopen if this is wrong. |
1 participant
Fixes #415
Summary
managed_microsimulationcould not run a local dataset file — only managed dataset names and remote URIs (hf://,gs://). A local build artifact (a downstream pipeline's per-year Stage-output H5 that is not part of any release manifest) raisedUnknown dataset, andallow_unmanaged=Trueonly relaxed URIs, not local paths (file://failed downstream withFileNotFoundError).This makes
resolve_managed_dataset_referenceaccept a local filesystem path whenallow_unmanaged=True— the same explicit opt-in already required for unmanaged URIs.materialize_dataset_sourcealready passes non-URI paths through unchanged, so the simulation constructs normally and records the provenance bundle (managed_by=policyengine.py). A local path withoutallow_unmanaged=Truenow raises an actionable error instead of the generic "Unknown dataset" message.Why
Local build-and-score pipelines — for example projecting the certified base to future years and then scoring reforms on the resulting local H5s — had to construct
policyengine_us.Microsimulationdirectly, bypassing the provenance recording and runtime-model pinning the managed path exists to enforce. They can now stay on the managed wrapper.Tests
test__given_local_path__then_managed_resolution_requires_opt_intest__given_local_path_with_opt_in__then_resolves_to_that_pathtest__given_unknown_dataset_name__then_raises_unknown_datasetVerified end-to-end:
managed_microsimulation(dataset="<local>.h5", allow_unmanaged=True)constructs a working simulation and records the provenance bundle. Fulltest_release_manifests.pypasses (33 tests);make lintclean.🤖 Generated with Claude Code