ESP32 integration suites exceed #806 900s cap on Windows; zccache persist_failed (tempdir nondeterminism) makes every framework rebuild cold

[zackees]

Summary

The #[ignore]d ESP32 integration suites (esp32_build, eh_frame_strip_esp32) cannot complete a single clean esp32dev framework build within #806's 900 s cap on this Windows machine — and the compile backend's zccache layer gives them no relief, for two independent reasons discovered while repairing the #800 backend-install gap (phase-8b PR, #1346):

1. Sequential framework builds blow the 900 s cap

The esp32dev framework stage compiles sequentially at roughly 4 s/file on this machine (Windows + xtensa-gcc). The Arduino-ESP32 core is ~1500 files → ~100 minutes per cold build, vs the 900 s budget both eh_frame_strip_drops_firmware_at_least_150kb (two builds!) and the four esp32_build blink/fixture tests need. Observed twice: timeout panic at exactly the 900 s cap, 938 s and 1055 s wall respectively.

2. zccache never hits for these runs — stores are refused

Embedded-zccache journals (~/.fbuild/prod/zccache/v1.13.1/logs/) show the esp32 framework compiles are effectively never cached:

journal window	hit	miss	error
rotation ending 2026-08-20 12:58	1	1761	91
rotation ending 2026-08-20 23:45	0	1838	2
current (2026-08-22 runs)	0	1558+	—

Two mechanisms in daemon-lifecycle.log:

• daemon_cache_root_contended — a test process's embedded backend does not own the cache-root writer slot whenever any other fbuild process touched it recently; its stores then can't be committed locally.
• persist_failed: "same cache key produced different staged output bytes" — the decisive one. Test builds run in a fresh tempfile::TempDir each invocation; the same content-keyed compile produces different output bytes across runs (almost certainly DWARF DW_AT_comp_dir/debug paths embedding the per-run tempdir), so zccache correctly refuses to persist them. Every subsequent run recompiles everything.

Real-daemon builds (stable project dirs) don't hit mechanism 2 — this is specific to fresh-tempdir test harnesses.

Impact

• All seven repaired-from-perf(zccache) Phase 4 stage 2: embedded mandatory — delete FBUILD_ZCCACHE_EMBEDDED + wrapper fallback #800 integration files now install a test backend; six go green (avr ×5, cache_survives ×3, clangd parity, teensy/nxplpc pending re-measure). The two ESP32 suites time out at the cap on Windows regardless of warm toolchain/cargo state.
• CI (Linux) may still fit the cap; these suites are #[ignore]d there anyway — so the gap is invisible until someone runs --include-ignored locally.

Suggested fixes (in dependency order)

1. Deterministic debug paths for cached framework compiles — -fdebug-prefix-map-style normalization (or drop -g for framework objects) so identical keys produce identical bytes and persist_failed stops eating every store.
2. Test-process cache isolation — route test-launched CompileBackends at a scratch/dev cache root (the way FBUILD_DEV_MODE=1 isolates), so tests neither contend for nor pollute the prod writer slot.
3. Parallelize the ESP32 framework stage (or revisit the sequential pipeline wall on Windows) — even fully cached-first-run builds will otherwise stay near the cap.
4. Optionally: have the orchestrator emit per-stage progress into the test-visible log so future timeouts are diagnosable without the deleted-tempdir problem noted in compile_many_stage2_perf oracle: unrunnable since #800; with backend installed, stage-2 wall ~6.5s fails 50%-of-stage-1 bound — #337 seed regression or stale baseline #1346.

Repro

soldr cargo test -p fbuild-build --test eh_frame_strip_esp32 -- --include-ignored
# → panicked: real-toolchain test exceeded 900s budget — see FastLED/fbuild#806

Found during phase-8b verification; cross-referenced from #1346.