feat(image): derive AVOCADO_OS_BUILD_ID from the assembled work tree

[nicksinas]

Fixes ENG-2441. Follow-up to ENG-2437 / #208.

Problem

The build id enumerated its inputs — the package NEVRA set, and in #208 the auth files — so anything off that list was invisible to the OTA gate, and the same bug recurred once per input (permissions, then overlay, then post_install). The list only grows.

Change

Replace the component list with a content hash of the assembled work tree, folded in beside the NEVRA hash:

AVOCADO_OS_BUILD_ID = uuid5(ns, "$PKG_HASH:$TREE_HASH")

via a shared render_build_id_block(&BuildIdSpec) so rootfs and initramfs can't diverge on the correctness-critical id logic. TREE_HASH moves iff the image bytes move → subsumes packages, permissions, overlay, post_install, and anything future.

What TREE_HASH hashes (and deliberately doesn't)

Covers exactly what the image carries — sorted path, type, mode (%m), symlink target (%l), and file content. Excludes:

• mtime and ownership — mkfs.erofs normalizes these out (-T "${SOURCE_DATE_EPOCH:-0}", --all-root), so hashing them would churn the id on noise the image never carries.
• directory sizes — fs-dependent.
• var/lib/rpm — the rpmdb sqlite embeds install timestamps (exactly why package identity is the NEVRA PKG_HASH, not these bytes). Pruned; PKG_HASH covers packages deterministically, including a version-only bump with an identical file payload.
• var/cache, var/log — dnf caches/logs, build-varying.

Ordering (a hard requirement, enforced by a test)

The hash is taken after permissions/post_install and before the os-release identity append, and the derivation strips the AVOCADO_* fields from the identity files first. Taking it after the append would make the id depend on itself and on the possibly-unpinned runtime version (runtimes.<name>.version defaults to a per-build uuid_v4() fragment — see ENG-2444) → an OTA on every build.

Prerequisite (same PR): HashMap → BTreeMap

PermissionsConfig.users/groups were a HashMap, whose per-process iteration order reshuffled the /etc/passwd appends and the auto-assigned UID/GIDs every build. With id = f(tree) that would churn the id every build, so this switches to BTreeMap (key-sorted, deterministic). No config-format impact — a BTreeMap deserializes from a YAML mapping identically.

Migration note

The id formula changes, so every existing image re-ids once on the first rebuild after this lands (a single OS OTA even if nothing else changed). Steady state is churn-free thereafter.

Tests

Unit tests: id = $PKG_HASH:$TREE_HASH; prunes present; no mtime/owner directives; strip → tree-hash → derive → append ordering; both images share render_build_id_block; BTreeMap provisions in sorted order. Full lib suite green; fmt/clippy clean (only the pre-existing Option::zip warning).

Empirically validated the exact GNU pipeline in a Linux container: determinism across two builds with differing mtimes/rpmdb bytes ✓, mtime ignored ✓, owner ignored ✓, rpmdb-churn pruned ✓, content/mode/symlink changes detected ✓.

⚠️ Pre-merge gate — determinism runbook (must run on a real SDK build/target)

Unit tests prove the script structure; only building twice proves the tree is reproducible. Draft until these pass:

1. Build-twice reproducibility (essential). clean && prune && install -f && build twice on the same machine → same AVOCADO_OS_BUILD_ID. If not, diff the two work trees (metadata via find … -printf '%y %m %P\t%l\n' | sort; content via find -type f | sort | xargs sha256sum). Prime suspects: usr/etc/ld.so.cache, systemd preset *.wants/* symlinks, usr/lib/opkg/alternatives, depmod modules.*. Fix at source or add to BUILD_ID_TREE_PRUNES.
2. Cross-machine / fresh checkout — same config → same id (catches host-leaking nondeterminism).
3. Permissions reorder is inert — reorder users:/groups: in YAML → id unchanged (BTreeMap check).
4. Sensitivity — each of {add a user, edit an overlay/ file, edit post_install, add/remove a rootfs.packages entry} → a different id.
5. On-device — unchanged deploy → device logs OS already at target version … skipping; permissions-only change → device downloads + applies the OS bundle.

I could not run 1/2 here (no SDK container/target).

Sequencing

Depends on nothing, but per the ticket: land #208, then ENG-2440 (stamp correctness), then this. Also relates to ENG-2444 (unpinned runtime version).

[Copilot]

Pull request overview

This PR updates the OS/initramfs build-id derivation to be based on a deterministic content hash of the assembled work tree (folded alongside the existing NEVRA-based package hash), so that any rootfs-affecting change (permissions, overlay, post_install, etc.) reliably changes AVOCADO_OS_BUILD_ID and therefore triggers OTA/deploy behavior correctly.

Changes:

• Switch permissions.users / permissions.groups to BTreeMap to guarantee deterministic provisioning order (and stable auto-assigned UID/GIDs).
• Add a shared render_build_id_block(&BuildIdSpec) used by both rootfs and initramfs to derive uuid5(ns, "$PKG_HASH:$TREE_HASH").
• Add/extend unit tests to enforce ordering and the build-id hashing/strip/append sequencing contract.

Reviewed changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated no comments.

Show a summary per file

File	Description
src/utils/permissions.rs	Rename helper to mapping_from_map, update to accept BTreeMap, and add a determinism/unit test for sorted provisioning order.
src/utils/config.rs	Change PermissionsConfig.users/groups from HashMap to BTreeMap and document why determinism is required.
src/commands/runtime/build.rs	Update call sites to use mapping_from_map after the config type change.
src/commands/rootfs/image.rs	Introduce render_build_id_block + tree-hash pruning and wire it into the rootfs image build script; add tests for structure and ordering.
src/commands/initramfs/image.rs	Reuse the shared build-id derivation block for initramfs and add a test asserting shared tree-hash usage and canonicalization.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[mobileoverlord]

Ran the pre-merge determinism runbook on a real SDK build box (qemux86-64, public 2026/edge feed, this branch's release binary):

• Item 1 — build-twice reproducibility: PASS. Full clean && prune && install -f && build twice → identical AVOCADO_OS_BUILD_ID (2554f067-9573-55f7-b390-88bd9e0d3ffd both cycles). Fresh installs between cycles means differing rpmdb/dnf bytes and mtimes throughout — all correctly invisible to the id.
• Item 4 — sensitivity: PASS. Adding one file under a rootfs.overlay dir moved the id (…3ffd → 04a38985-…). This also validates the fix(stamps): hash overlay contents for verbatim overlays too #210 + this-PR interplay end to end: overlay content now reaches both the install stamp and the OTA gate.

Not run: item 2 (cross-machine — one box only), item 5 (on-device OTA). Item 3 (permissions reorder) is subsumed by item 1 under BTreeMap.

Also, changes made while rebasing this over today's merges (#203/#204/#208 landed underneath it):

• render_auth_files_hash/AUTH_HASH is now fully subsumed and deleted — the ENG-2437 regression is re-expressed in tree-hash terms (permissions section ordered before the tree hash, /etc never prunable).
• ./var/lib/dnf added to BUILD_ID_TREE_PRUNES — history.sqlite changes bytes on every install transaction, which under id = f(tree) is an OTA on every build.
• The package-state purge now runs before the id derivation (test-pinned), so the tree hash covers exactly what ships.

[mobileoverlord]

Design matches ENG-2441 on every hard requirement (shared render_build_id_block, hash before identity append with the AVOCADO_* strip test-pinned, BTreeMap prerequisite, mtime/ownership excluded to match mkfs.erofs). The PKG_HASH retention alongside TREE_HASH is the right deviation given the rpmdb prune. Determinism runbook items 1 and 4 pass on a real SDK build (evidence in comments). One migration note stands: every existing image re-ids once on first rebuild after this lands.