Skip to content

sealing blog#40

Draft
jeckersb wants to merge 4 commits intobootc-dev:mainfrom
jeckersb:sealing-blog
Draft

sealing blog#40
jeckersb wants to merge 4 commits intobootc-dev:mainfrom
jeckersb:sealing-blog

Conversation

@jeckersb
Copy link
Copy Markdown
Collaborator

@jeckersb jeckersb commented May 1, 2026

  • blog: add sealed images security chain overview
  • blog: add sealed images key management post
  • blog: add sealed images building post
  • blog: add sealed images deploying post

Copy link
Copy Markdown
Contributor

@gemini-code-assist gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces a four-part blog series detailing the security chain, key management, build process, and deployment of sealed images for bootc. The review feedback identifies technical inaccuracies regarding bootupd's support for systemd-boot and the range of filesystems supporting fs-verity. Additionally, a correction was provided for a Dockerfile example that was missing the necessary steps to install the generated Unified Kernel Image (UKI) into the boot directory.

Comment on lines +113 to +118
bootc container ukify --rootfs /target \
-- \
--signtool sbsign \
--secureboot-private-key /run/secrets/secureboot_key \
--secureboot-certificate /run/secrets/secureboot_cert \
--output /out/uki.efi
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

The code block is missing the command to move the generated UKI to the expected path in /boot. The text at line 138 mentions that the stage places the UKI under /boot/EFI/Linux/, but the ukify command outputs to /out/uki.efi. This will cause the subsequent COPY --from=kernel /boot /boot in Stage 4 to miss the new UKI.

Suggested change
bootc container ukify --rootfs /target \
-- \
--signtool sbsign \
--secureboot-private-key /run/secrets/secureboot_key \
--secureboot-certificate /run/secrets/secureboot_cert \
--output /out/uki.efi
bootc container ukify --rootfs /target \
-- \
--signtool sbsign \
--secureboot-private-key /run/secrets/secureboot_key \
--secureboot-certificate /run/secrets/secureboot_cert \
--output /tmp/uki.efi
mkdir -p /boot/EFI/Linux/
install -m 0644 /tmp/uki.efi /boot/EFI/Linux/sealed-uki.efi

Comment on lines +50 to +51
filesystem. This is required because fs-verity is only supported
on ext4 and btrfs.
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

The statement that fs-verity is only supported on ext4 and btrfs is outdated. Support for fs-verity on XFS was merged in Linux kernel 6.10. Since the example uses CentOS Stream 10 (which uses kernel 6.11+), XFS is also a valid option.

Suggested change
filesystem. This is required because fs-verity is only supported
on ext4 and btrfs.
filesystem. This is required because fs-verity is supported
on ext4, btrfs, and XFS (in modern kernels).

Copy link
Copy Markdown
Contributor

@cgwalters cgwalters left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks so much for doing this! Only glancing at the first one so far.

Also once we publish these we should cross-link them from the bootc composefs docs.


## The gap

Many Linux systems in use today don't use
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks similar to some other content I helped edit and would like to change, see https://url.corp.redhat.com/1779017

bootloader may verify the kernel's signature, but the initramfs and
command line are typically not covered by any signature.

A Unified Kernel Image (UKI) changes this by bundling the kernel, the
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Worth linking to the spec here I think

line is covered by the same signature that covers the kernel itself.
As we'll see next, this is where the composefs digest lives.

## The composefs digest
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would probably refactor this to say:

## composefs: EROFS + overlayfs + fsverity

And frame it that way, e.g. the section here is saying that the digest is of the EROFS...

carries within it a cryptographic commitment to the exact filesystem
that should be mounted as the operating system.

## fs-verity
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

And this section could explain briefly not just fsverity but also the overlayfs part.

@bootc-dev bootc-dev deleted a comment from gemini-code-assist Bot May 1, 2026
### Check the root mount

```
$ mount | grep " / "
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AI: Important: This command will not match the output shown. Grepping for " / " (with spaces) will not match a line containing /sysroot. Additionally, standard mount output is <device> on <mountpoint> type <fstype> (options) — the output shown does not match that format.

In a booted bootc system the composefs/overlayfs mount enforcing verity=require will be at the deploy root, not necessarily at /sysroot. A more reliable command would be:

$ findmnt -t overlay /

or

$ mount | grep -E "composefs|overlay"

with output updated to reflect what these actually produce. Readers on Reddit/HN will spot fake terminal output immediately.

Comment on lines +152 to +153
built. Configuration management and runtime security tools protect
the rest.
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This starts to get into bigger picture design, but for /etc i think we should actually strongly encourage transient or readonly even.

And for /var, this is where PCR-locked LUKS can come into play (and/or dm-integrity).

How about for now:

By default, /etc and /var are mutable and persistent. It is possible to make /etc transient today, and this is a good idea if it matches your use case. More suggestions on this will be in the documentation.

@jeckersb jeckersb force-pushed the sealing-blog branch 2 times, most recently from 2a0678b to 572134c Compare May 5, 2026 18:39
@jeckersb
Copy link
Copy Markdown
Collaborator Author

jeckersb commented May 5, 2026

1. Broken link in /github/workspace/content/blog/2026-apr-28-sealed-images-key-management.md to https://www.freedesktop.org/software/systemd/man/latest/loader.conf.html : Client error status code (418 I'm a teapot) received

due to anti-bot but does work if I visit manually in my browser. Will amend the link checker to skip with a comment.

@jeckersb
Copy link
Copy Markdown
Collaborator Author

jeckersb commented May 5, 2026

AI feedback that I'll post here and then feed back into the AI to fix it...

Review Summary
The series is well-written and technically solid overall. Most of the previous review feedback from cgwalters and gemini-code-assist has been addressed in the current revision (the composefs section was restructured, UKI spec link added, /etc//var scope text updated, findmnt used instead of mount | grep, etc.). However, several issues remain or are newly identified:
Issues Found
1. fs-verity filesystem support is incorrect (deploying post, line ~51)
The current text says:
> fs-verity is supported on ext4, btrfs, and XFS (on modern kernels).
This is wrong. Checking the Linux kernel source (fs/verity/Kconfig and Documentation/filesystems/fsverity.rst), fs-verity is supported on ext4, btrfs, and f2fs only. There is zero XFS fs-verity support in the kernel -- no FS_VERITY references anywhere in fs/xfs/. The gemini-code-assist suggestion about XFS was incorrect, and it appears to have been accepted. This should be reverted to say "ext4 and btrfs" (f2fs is not relevant for bootc), or more precisely: "ext4, btrfs, and f2fs" with a note that f2fs is not typically used for root filesystems.
2. The --filesystem=ext4 explanation is misleading (deploying post, line ~50-51)
The text says this flag is "required because" of fs-verity support limitations. But the flag is specifying which filesystem to use, not asserting a requirement. The default for bootc install may be XFS (which doesn't support fs-verity), so the flag is needed to choose a verity-capable filesystem. The framing should clarify this -- something like "The default filesystem may be XFS, which does not support fs-verity, so we explicitly select ext4."
3. Stage 3 Containerfile snippet is still incomplete (building post, lines ~113-125)
The current diff shows Stage 3 as:
FROM base AS kernel
RUN --mount=type=bind,from=base,target=/target \
    --mount=type=secret,id=secureboot_key \
    --mount=type=secret,id=secureboot_cert <<EOF
bootc container ukify --rootfs /target \
    -- \
    --signtool sbsign \
    --secureboot-private-key /run/secrets/secureboot_key \
    --secureboot-certificate /run/secrets/secureboot_cert \
    --output /out/uki.efi
EOF
RUN kver=$(bootc container inspect --json | jq -r '.kernel.version') && \
    mkdir -p /boot/EFI/Linux && \
    mv /out/uki.efi "/boot/EFI/Linux/${kver}.efi"
The second RUN step that installs the UKI to /boot/EFI/Linux/ is present in the current version, which addresses gemini's original concern. However, bootc container inspect is being called -- this should be verified as a real subcommand. Looking at the bootc CLI source (cli.rs), the container subcommands are lint, ukify, and inspect -- so bootc container inspect --json does exist. This looks correct.
4. virt-firmware URL inconsistency (key management vs deploying post)
In the key management post, virt-firmware links to https://gitlab.com/kraxel/virt-firmware. In the deploying post, it links to https://github.com/rhuefi/virt-firmware. These should be consistent -- verify which is the canonical upstream and use that one in both places.
5. The secureboot-keys path description (key management post, line ~210)
The post says files should be placed at /usr/lib/bootc/install/secureboot-keys/<name>/. The bootc source code (boot.rs:139) confirms the path is usr/lib/bootc/install/secureboot-keys, and it expects subdirectories containing .auth files. The blog text is accurate here.
Minor/Nit
6. SHA-512 vs SHA-256 for the composefs digest
The posts consistently say "128-character SHA-512 hex digest" for the composefs digest. From the bootc source, the digest is computed via compute_image_id() which calls compute_verity() on the EROFS image bytes. The composefs-rs code supports both SHA-256 and SHA-512 -- the actual algorithm used depends on how the FileSystem is parameterized. The blog should ideally note which algorithm bootc uses by default rather than stating SHA-512 as a fact, unless you've confirmed bootc always uses SHA-512 for the image ID. (A 128-char hex string = 64 bytes = SHA-512, so if that's what you see in practice, it's fine -- just worth verifying.)
7. The open() succeeds / read() returns EIO description (security chain post, lines ~114-117)
This is technically accurate for regular fs-verity, but worth noting that for composefs with verity=require, the overlayfs layer checks the fs-verity digest at open() time too (via metacopy verification). The current description is close enough for a blog post audience though.
Already Addressed (from prior reviews)
- cgwalters' suggestion to restructure as "composefs: EROFS + overlayfs + fs-verity" -- done
- cgwalters' suggestion about UKI spec link -- done
- cgwalters' /etc//var scope text -- done
- cgwalters' findmnt instead of mount | grep -- done
- link_checker skip for freedesktop.org -- done
Bottom Line
The critical fix needed is item #1: XFS does not support fs-verity. This is a factual error that will be noticed by kernel-savvy readers. Items #2 and #4 are important but lower priority.

jeckersb added 4 commits May 5, 2026 15:00
First post in a series about bootc sealed images, covering the
full trust chain from Secure Boot firmware through per-file fs-verity
verification at runtime.

Assisted-by: OpenCode (claude-opus-4-6)
Second post in the sealed images series, covering the UEFI Secure
Boot key hierarchy, key generation, signing boot artifacts, and
key enrollment methods.

Also adds a link_checker skip for freedesktop.org which returns 418
to bots.

Assisted-by: OpenCode (claude-opus-4-6)
Third post in the sealed images series, covering the multi-stage
container build process for producing a sealed, signed bootc image.
References the rhel-bootc-examples repository for the complete
working example.

Assisted-by: OpenCode (claude-opus-4-6)
Fourth post in the sealed images series, covering deployment via
bcvk and verification that the full security chain is active.

Assisted-by: OpenCode (claude-opus-4-6)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants