Simulation cache is write-only: uuid4 keys mean ensure() never reloads results, and data_folder accumulates orphaned h5 files

[MaxGhenis]

Symptom

The simulation result cache is write-only across processes: identical reruns recompute everything, and data_folder silently accumulates orphaned output files.

Observed after one evening of normal use (three microsimulation scripts over a 2026–2035 window on the certified national dataset, policyengine 4.18.10):

• ./data contained 71 orphaned <uuid>.h5 output files (2.0 GB) alongside the 10 reusable populace_us_2024_year_*.h5 dataset files
• rerunning the identical script recomputed all 20 simulations (~64 minutes) instead of loading them

Mechanics

1. Simulation.id defaults to a random uuid4 (src/policyengine/core/simulation.py:57).
2. Simulation.ensure() checks the in-memory cache with _cache.get(self.id) (simulation.py:133) — only the same object can ever hit, since no two Simulation objects share an id.
3. The disk path is keyed on the same id: load() looks for <dataset_dir>/<simulation.id>.h5 (src/policyengine/tax_benefit_models/common/model_version.py:316). A fresh Simulation in a new process (or even the same process) can never name a previous run's file, so ensure() always falls through to run() + save() — which writes yet another <uuid>.h5 that nothing will ever read.

Net: ensure()'s documented behavior ("loads a cached result if one exists, or runs and caches on miss" — docs/microsim.md) only holds within the lifetime of a single Simulation object.

By contrast, ensure_datasets keys year-uprated datasets deterministically ({stem}_year_{year}.h5, src/policyengine/tax_benefit_models/us/datasets.py) and reuses them correctly across runs — the pattern the simulation layer is missing.

Second-order: the in-memory LRU pins whole simulations

ensure() ends with _cache.add(self.id, self) — the entire Simulation object, including its output_dataset, into a count-bounded LRU (max_size=100, src/policyengine/core/cache.py). In a multi-year loop this pins every year's outputs in RSS long after the loop variables go out of scope; we watched RSS climb steadily across a 10-year run for this reason. The memory warning it emits is also misleading: it reports process-wide RSS ("Memory usage has reached 32.77GB… Cache contains 1 items") — the RSS is dominated by the country simulation internals, not the one cached item.

Proposed design

1. Content-addressed cache key. A stable digest over what actually determines the result:
   • dataset identity (logical name + year + data build/version, e.g. the release-bundle certified_data_build_id for managed datasets, or file digest for custom ones)
   • tax-benefit model package + version
   • policy and dynamic parameter values, canonicalized (name, start, stop, value)
   • scoping strategy and resolved entity/extra variables
     Expose as Simulation.cache_key; keep the uuid4 id for run identity/provenance (TRO run records keep their own composition_fingerprint).
2. Key save()/load() and the in-memory LRU on cache_key instead of id. Same-inputs reruns then load in seconds, across objects and processes.
3. Bound the in-memory cache by what it holds. Cache output_dataset rather than the whole Simulation, with a much smaller default (or byte-based bound), so multi-year loops don't accumulate RSS.
4. Garbage collection. With deterministic keys, orphans stop being produced; a policyengine cache clear/prune for existing UUID orphans (or prune-unreadable-on-run) cleans up installations that already have gigabytes of them.

Impact

Multi-year revenue scoring (the 10-year budget window use case) currently costs a full recompute per invocation — ~2.5 hours for a 3-scenario × 10-year set that a functioning cache would serve in seconds after the first run — plus ~2 GB/evening of disk litter.

Related: #328 (example memory footprint — the LRU pinning above is one contributor), #320 (local reproducibility snapshots — a content-addressed key is also the natural handle for those).

Versions: policyengine 4.18.10, Python 3.13/3.14, macOS.