Skip to content

initrd/bin/kexec-sign-config.sh: stop re-hashing stale default entry paths on -u update#2150

Open
tlaurion wants to merge 1 commit into
linuxboot:masterfrom
tlaurion:fix-stale-rehash-kexec-sign-config
Open

initrd/bin/kexec-sign-config.sh: stop re-hashing stale default entry paths on -u update#2150
tlaurion wants to merge 1 commit into
linuxboot:masterfrom
tlaurion:fix-stale-rehash-kexec-sign-config

Conversation

@tlaurion

@tlaurion tlaurion commented Jul 6, 2026

Copy link
Copy Markdown
Collaborator

Remove the -u block that re-hashed file paths extracted from the old kexec_default_hashes.txt. Those paths are from a previously saved default boot entry. If a distro replaced a versioned boot file in-place (e.g. Qubes OS bumping xen-4.19.4.gz to xen-4.19.5.gz), the old path no longer exists on disk and sha256sum fails, aborting the entire update with DIE.
Read: delete old binaries at the same time of providing new ones, which never happened before.

Qubes OS does NOT fan out old and new xen binaries on /boot: a dom0 update replaces xen-4.19.4.gz with xen-4.19.5.gz in-place (QubesOS/qubes-vmm-xen#220, shipped to stable via QubesOS/updates-status#6746 on 2026-07-02). Other distros (Fedora, Arch, Debian, NixOS, Ubuntu...) keep old kernels+initramfs alongside new ones, so the old paths still resolve and the re-hash failure was never observed there.

The re-hash had been present since its introduction in 2018 but went unnoticed because most distros keep old kernel/xen files on disk alongside new ones, so the old paths still resolved.

The failure is now fatal because the -u (update) and -r (rollback counter) flags are passed together in a single call. When the re-hash fails, DIE is called before the rollback counter is ever incremented, leaving the system with a partially updated kexec_hashes.txt but no counter bump. This coupling was introduced in commit 8bfae52 (Oct 2020), which restructured update_checksums to always call kexec-sign-config with -u hardcoded and -r as extparam.

Regression lineage:

  • Mar 14, 2018 - @kylerankin introduced the re-hash in update_checksums() inside gui-init (commit 769f6a7).
  • Jul 5, 2019 - @MrChromebox moved update_checksums() to /etc/functions (commit ed2f19d).
  • Oct 8, 2020 - @alex-nitrokey moved the re-hash block into kexec-sign-config under the -u flag and coupled it with -r (rollback) in the same call (commit 8bfae52).
  • Through 2026 - No OS removed boot binaries atomically without leaving old files in place, so the coupled failure went untriggered.

Mitigation on unpatched firmware (recovery shell):

mount -o rw,remount /boot
rm /boot/kexec_default_hashes.txt
mount -o ro,remount /boot
reboot.sh

Then re-run Update checksums. The -e guard skips the re-hash when the file is gone. Then set a new default from the boot menu.

Fix: skip the re-hash entirely. The user re-saves the default boot entry from the boot menu to regenerate kexec_default_hashes.txt.


Was discussed over matrix's Dasharo Premier support channel (subscribers only) here.

Affects all Heads + QubesOS 4.3 users

Copilot AI review requested due to automatic review settings July 6, 2026 00:00
@tlaurion tlaurion added bug feature freeze - 2025-01-30 (originally targeted 2024-11-20) labels Jul 6, 2026

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot wasn't able to review any files in this pull request.


💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

@tlaurion

tlaurion commented Jul 6, 2026

Copy link
Copy Markdown
Collaborator Author

QubesOS automated testing was testing (results still available as writing)

Note that replacing xen-4.19.4.gz with xen-4.19.5.gz landed July 2nd, as per normal testing->release falling through all current automated testing in place for QubesOS QA/Dasharo.

CC: @marmarek @macpijan @pietrushnic, let me know how we want to handle this forward.

  • One possibility is for QubesOS to drive emergency Xen binary update, not removing old one so grub.cfg entries are regenerated leaving user the possibility to revert older binary from availabe boot entries (as for current kernel/initramfs)
  • Release a bugfix release for all platforms using Qubes (Novacustom/Nitrokey/Purism/HardenedVault/Others/All)
  • Push mitigation as detailed in OP. No impact known for any other OSes since 2018; typically OS updates installs new kernel/initramfs do not force deletion of old ones "atomically": this is left for a cleanup process which respects the number of rollback of boot entries defined.

@tlaurion tlaurion force-pushed the fix-stale-rehash-kexec-sign-config branch 2 times, most recently from 8fe881f to 7acbb08 Compare July 6, 2026 00:54
…paths on -u update

Remove the -u block that re-hashed file paths extracted from the old
kexec_default_hashes.txt.  Those paths are from a previously saved
default boot entry.  If a distro replaced a versioned boot file
in-place (i.e. atomically deletes old binaries while providing new
ones), the old path no longer exists on disk and sha256sum fails,
aborting the entire update with DIE.

Qubes OS is the first distro that replaces boot binaries atomically
without fan-out: a dom0 update replaces xen-4.19.4.gz with
xen-4.19.5.gz in-place (QubesOS/qubes-vmm-xen#220, shipped to stable
via QubesOS/updates-status#6746 on 2026-07-02).  Other distros
(Fedora, Arch, Debian, NixOS, Ubuntu) keep old kernels alongside new
ones, leaving deletion to an OS cleanup policy, so the old paths
still resolve and the re-hash failure was never observed there.

The re-hash had been present since its introduction in 2018 but went
unnoticed because no OS had ever removed boot binaries atomically
without leaving the old files in place — Qubes OS automated testing
on heads commit 426750a (2026-04-22, pre_release_dasharo_101_rc3)
passed because xen 4.19.4 was still current and the old paths
resolved (openQA tests T430 openqa.qubes-os.org/tests/187218,
V560TU openqa.qubes-os.org/tests/186420).  It became blocking only
after xen 4.19.5 replaced 4.19.4 on disk, at which point any user
with a default boot entry pointing at xen-4.19.4.gz would hit the
fatal failure.

The failure is now fatal because the -u (update) and -r (rollback
counter) flags are passed together in a single call.  When the re-hash
fails, DIE is called before the rollback counter is ever incremented,
leaving the system with a partially updated kexec_hashes.txt but no
counter bump.  This coupling was introduced in commit 8bfae52
(Oct 2020), which restructured update_checksums to always call
kexec-sign-config with -u hardcoded and -r as extparam.

Regression lineage:
- Mar 14, 2018  - Kyle Rankin introduced the re-hash in update_checksums()
                  inside gui-init (commit 769f6a7).
- Jul  5, 2019  - Matt DeVillier moved update_checksums() to /etc/functions
                  (commit ed2f19d).
- Oct  8, 2020  - alex-nitrokey moved the re-hash block into kexec-sign-config
                  under the -u flag and coupled it with -r (rollback) in the
                  same call (commit 8bfae52).
- Through 2026  - No OS had removed boot binaries atomically without leaving
                  old files in place, so the coupled failure went untriggered.

Mitigation on unpatched firmware (recovery shell):
  mount -o rw,remount /boot
  rm /boot/kexec_default_hashes.txt
  mount -o ro,remount /boot
  reboot.sh
Then re-run Update checksums.  The -e guard skips the re-hash when the
file is gone.  Then set a new default from the boot menu.

Fix: skip the re-hash entirely.  The user re-saves the default boot
entry from the boot menu to regenerate kexec_default_hashes.txt.

Signed-off-by: Thierry Laurion <insurgo@riseup.net>
@tlaurion tlaurion force-pushed the fix-stale-rehash-kexec-sign-config branch from 7acbb08 to 3950b73 Compare July 6, 2026 00:54
@tlaurion tlaurion requested a review from Copilot July 6, 2026 00:59

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot wasn't able to review any files in this pull request.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug feature freeze - 2025-01-30 (originally targeted 2024-11-20) upstream

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants