[Draft] Migrate to mkosi + sysupdate

[jumpyvi]

Migrate to mkosi and sysupdate

I got basic boot, missing a few package update to be able to check the full experience with Resolute.

Fixes settings-daemon/#167, #724 and #717

[nikodunk]

This is more elegant than #808, and builds out the mkosi/sysupdate method from #793 that fully supports ubuntu already. In comparison to my bootc suggestion, this:

• Build: mkosi build --profile=sysupdate Create disk image (currently .raw but could be a fake .iso).
• Install: rsync mkosi.output/elementary.raw https://r2.cloudflarestorage.com/installer/x86.iso Copy the image to the download link, so users can download it and write to a USB drive. Live boot session would need to copy this image to disk. Installer would need to be adapted.
• Update: rsync mkosi.output/* https://r2.cloudflarestorage.com/updates Push image to any static host so all existing ElementaryOS installs can run updatectl update (settings app would need to be adapted). This checks a static folder at ie. updates.elementary.io (currently set to 10.0.2.2:7676 for testing), downloads anything newer than the currently-booted IMAGE_VERSION, drops it into the inactive A/B slot, and reboots into it on next boot. Even over major versions like 8=>9=>10.

[nikodunk]

@danirabbit what's the best way to land this? This PR currently replaces the old stuff, but I'm guessing some the 8.1 stable jobs might be run in parallel with the 9.0 sysupdate jobs until 8.1 reaches EOL?

Edit: My proposal, not speaking for @jumpyvi here:

• When ready to merge, make current os/main branch into archive and keep running the 8.1 stable jobs off of that archive branch.
• This becomes main, and starts building the sysupdate isos and updates.

[nikodunk]

Very nice.

[jumpyvi]

This is a local server for now, CDN/buckets are more expensive than I tough

It can be started with just sign-repo and just serve

If you use the flatpak version of qemu, the host will always be accessible at 10.0.2.2

[danirabbit]

@lewisgoddard manages our CDN when we want to look at actually serving this somewhere

[danirabbit]

@nikodunk yeah I think we probably want to make sure not to break existing build jobs until we're shippable, including work in installer, initial setup, system settings, whatever server infrastructure we need, etc.

AFAICT it doesn't look like this branch yet includes build and publish workflows? We probably want the daily publish workflow set up for these so that they're accessible from the builds website and we can easily verify future PRs don't break builds

[danirabbit]

Sysupdate is still missing delta updates too right? So I wonder if we want to keep the old daily workflow regardless and only use these for stable builds until we have deltas so we're not driving massive infra costs

[jumpyvi]

Sysupdate is still missing delta updates too right? So I wonder if we want to keep the old daily workflow regardless and only use these for stable builds until we have deltas so we're not driving massive infra costs

Yes, it could also get pretty expensive

Zirconium solution to this is to use Oras, but I don't this is a viable production-ready solution https://github.com/tulilirockz/pompona

[nikodunk]

OK so going back on my above suggestions, how about we keep sysupdate on a branch, and add a 4th job on a sysupdate branch then:

• daily-8.1 (main branch)
• daily-9.0 (main branch)
• stable-8.1 (main branch)
• image-9.0 (sysupdate branch this can be a manual kick-off, and not advertised until there's delta updates?)

[jumpyvi]

* image-9.0 (sysupdate branch this can be a manual kick-off, and not advertised until there's delta updates?)

We don't know when/if systemd will choose to implement this.

idk if there is any other solutions

systemd/systemd#28227

[nikodunk]

So back of the napkin: a 4gig image, hosted on hetzner s3, downloaded once a month for 10,000 monthly active users, would be 40K gigs a month, or 40tb monthly traffic roughly = $54/mo

Is that a dealbreaker for this new image-based system @danirabbit ? Bc if so, maybe we should move back into bootc territory (where oci hosting is an order of magnitude cheaper through github OCI and delta updates are supported) even though the systemd ecosystem is nicer for debian/ubuntu.

Prior art:

• Gnome OS powers through the cost of updates
• KDE linux implemented delta updates themselves https://community.kde.org/KDE_Linux/Delta
• For Zirconium @tulilirockz has a workaround where she ships a DDI in an OCI container, but there's a little glue involved.
• importd: add support for downloading OCI images systemd/systemd#39621

Sorry, @jumpyvi and i were just chatting on discord and want to make sure we're not painting ourselves into a technical corner here :D

[danirabbit]

@nikodunk I don't think that would be a huge problem no. If it were like $1k/mo that would be an issue but $50–100 I think we can deal with

[lewisgoddard]

For S3 compatible storage I would recommend Cloudflare R2. We already use Cloudflare and there is no bandwidth billing there, only storage and action costs.

[nikodunk]

Thank you! Looks like we were holding it wrong: cost won’t be a problem at $0.60 for 10,000 users downloading an 4gb image once a week.

OK so maybe @jumpyvi we don't optimize for transmission size yet for now then, and optimize for deltas later as sysupdate matures - other projects will have this problem too.