RFC: Kerf Shared Memory (`kerf shm`) Design

[congwang-mk]

Motivation

Today a daxfs image is welded to a single instance: kerf load --image/--rootfs-dir
builds it, keys it by instance name, and passes its physical address on that
instance's cmdline. Sharing memory between instances means manually recovering
a physical address, and there is no CLI surface for daxfs as anything other
than a root filesystem.

This design promotes shared memory to a first-class kerf object. A daxfs is
the presentation of shared memory: two instances that mount the same daxfs and
mmap the same file share physical memory directly (DAX, no page cache),
exactly as POSIX shared memory is files on tmpfs within one kernel. A future
unformatted "raw" kind covers consumers that want a bare region.

Concepts

shm object: a named, host-managed region allocated from
/dev/dma_heap/multikernel, existing independently of any instance, with a
kind:

• fs (this design): formatted as daxfs, host-mounted, visible in every
  attached instance at /shm/<name>.
• raw (future, out of scope): unformatted region announced to the guest by
  physical address only. Deferred until a kernel-side pin for unformatted
  dma-heap allocations exists (for fs, the host daxfs mount holds the
  dma-buf reference; raw has no mount to do that).

The metadata carries kind from day one so raw slots in without breaking
anything.

Backing: where the bytes come from is orthogonal to the kind and to
attachment, which only ever sees name plus (phys, size). v1 supports two:

• dmabuf (default): allocated from /dev/dma_heap/multikernel. Writable
  mapping comes from mmap'ing the dma-buf fd; the host daxfs mount holds the
  dma-buf reference (pin); rm frees by unmounting.
• cma: a named allocation via /dev/lazy_cma under the iomem name
  kerf-shm:<name>, optionally NUMA-placed. The allocation lives in the
  kernel resource tree with no fd to keep alive, so it is pinned by nature
  (this is also exactly the pin the future raw kind needs, which is why
  raw becomes straightforward once cma backing exists). Requires two small
  lazy_cma module extensions: mmap (so kerf can write the daxfs image into
  the region) and a free ioctl (so rm can return the memory). Until those
  land, kerf reports a clear error for --backing=cma.
• extern (future, out of scope): adopt any pre-existing phys+size region
  (boot reservation, NVDIMM). kerf validates and registers but never
  allocates or frees.

The host daxfs mount works for every backing: dmabuf uses the existing
FSCONFIG_SET_FD path, everything else uses daxfs's phys=/size= options,
the same ones guests already use in rootflags=.

Rootfs is not an shm. Root filesystems stay per-instance and private via
the existing --image/--rootfs-dir flow. Sharing happens exclusively
through /shm/<name> mounts; a shared read-only base rootfs, if ever needed,
is a separate future feature.

Sharing semantics: concurrent multi-writer, provided by daxfs in the
multikernel kernel tree (cross-kernel coordination in the overlay region;
format bump). That work is a dependency, not part of this design. Until it
lands, kerf warns when attaching an shm that already has a live attachment.

CLI

kerf shm create <name> --size=SZ [--overlay-pool=SZ] [--backing=dmabuf|cma] [--node=N]
kerf shm create <name> --from-dir=DIR [--overlay-pool=SZ] [--backing=dmabuf|cma] [--node=N]
kerf shm ls
kerf shm show <name>
kerf shm rm <name> [--force]
kerf shm attach <name> <instance>
kerf shm detach <name> <instance>

• --size alone creates an empty writable daxfs (empty root directory; the
  size beyond fixed structures goes to the overlay pool).
• --from-dir prepopulates from a directory, reusing the existing builder.
  (No --from-image: container images are a rootfs concern; extract to a
  directory first if ever needed.)
• --backing defaults to dmabuf; --node (NUMA placement) is valid only
  with --backing=cma, since the dma-heap allocation ioctl has no node
  parameter.
• Names: [a-z0-9][a-z0-9_-]*, unique, distinct namespace from instances,
  and short enough that kerf-shm:<name> fits lazy_cma's 63-byte iomem
  resource name limit.
• rm refuses while attachments exist unless --force (which only clears
  bookkeeping; it does not yank memory from running instances, so it warns).

Attachment works at load time (repeatable --shm=<name> flag) and at
runtime, both reducing to the same mechanism:

kerf load web    --kernel=... --image=nginx:latest --shm=data
kerf shm attach logs web       # while web is running (or loaded, pre-boot)
kerf shm detach logs web

Convention over configuration: each attached shm appears in the guest at
/shm/<name>, mounted by kerf-init. No per-attachment mountpoints or modes;
a workload that needs content at a specific path symlinks or bind-mounts
inside the guest.

attach/detach update metadata immediately. Against a running instance
they wait (bounded) for the guest to acknowledge and report its actual
outcome (mounted, busy, error); against a loaded-but-not-running instance
they apply at next boot. detach of a busy mount fails in the guest and is
reported as such; it does not force-unmount.

Control channel and guest discovery

No user ever handles a physical address; kerf resolves names to (phys, size)
from its metadata and there are no per-attachment cmdline parameters. The
control channel is the multikernel vsock transport (linux commit
7b3d04bf133e53f2 "vsock: Introduce multikernel transport"): AF_VSOCK over
multikernel IPI messaging, CID = instance id with the host at CID 0, opt-in
per socket via SO_VM_SOCKETS_TRANSPORT = VSOCK_TRANSPORT_MULTIKERNEL.
Reliable connection-oriented streams, interrupt-driven, standard socket API
on both ends.

• kerf-init listens on a well-known port (KERF_AGENT_PORT, e.g. 1) inside
  the guest.
• The host side is always a short-lived kerf CLI process connecting to
  (CID = instance id, KERF_AGENT_PORT); no host daemon.
• Protocol: versioned, struct-packed request/response messages.
  SET_SHM carries the full desired attachment set, entries of
  (name, phys, size); the response carries per-name outcomes (mounted,
  unmount-busy, error). Sending the complete set makes the guest side a
  reconciler: idempotent, order-independent, and correct on replay.

Boot sequencing: since the host cannot push before the guest listener
exists, kerf exec performs the initial push. kerf-init starts its
listener, then waits for the first SET_SHM before starting the
entrypoint; kerf exec triggers boot, connects with retry/backoff, sends
the desired set from metadata (possibly empty), and reports the guest's
mount status. The entrypoint therefore never starts before its /shm/
mounts are in place, and a failed boot-time mount fails kerf exec loudly
(fail-fast). This also makes replay after kerf kill + kerf exec
automatic, since exec always pushes the current desired set.

kerf shm attach/detach at runtime are the same SET_SHM push over a
fresh connection, after updating metadata.

The rootfs cmdline (rootfstype=daxfs rootflags=phys=...,size=...) is
unchanged.

kerf-init as resident PID 1

kerf-init stops exec'ing the entrypoint and instead: brings up the vsock
listener, receives the initial SET_SHM, reconciles mounts under /shm/,
replies with status, forks the entrypoint, then remains resident as PID 1,
reaping children and serving attach/detach requests for the lifetime of the
instance. Runtime failures (e.g. detaching a busy mount) are reported in the
response and never force-unmount.

Host-side storage and metadata

Mirrors the instances layout:

/var/lib/kerf/shm/<name>.json   # metadata, atomic write via os.replace
/var/lib/kerf/shm/<name>/       # host daxfs mountpoint (pins the dma-buf)

Metadata schema:

{
  "name": "data",
  "kind": "fs",
  "backing": "dmabuf" | "cma",
  "node": -1,
  "phys_addr": 123456789,
  "size": 2147483648,
  "created": "2026-08-15T10:00:00Z",
  "source": {"type": "empty" | "dir", "path": "..."},
  "attachments": ["web", "worker"]
}

• Instance metadata (/var/lib/kerf/instances/<name>.json) records attached
  shm names symmetrically under a "shm" key, following the existing
  provenance pattern, so kerf show <instance> needs no cross-lookup.
• Staleness: shm memory does not survive host reboot but the json files do.
  ls, show, and attachment resolution verify the recorded region still
  appears in /proc/iomem (as daxfs for dmabuf backing, as
  kerf-shm:<name> for cma backing, where the exact name makes the check
  precise); dead entries are reported as stale by ls and pruned on rm or
  on the next create of the same name.
• Physical address discovery is per backing: lazy_cma returns the address
  from its alloc ioctl directly, while dmabuf keeps today's
  snapshot-and-diff of daxfs regions in /proc/iomem.
• Attachment bookkeeping updates on kerf load and kerf shm attach (add),
  kerf shm detach, kerf unload and kerf delete (remove). kerf kill
  leaves attachments in place, matching its existing "instance still loaded"
  semantics; they are reconciled again at next boot.

The host mount is not just a pin: with multi-writer daxfs it is the host's
live window into the filesystem (drop a file in from the host, a running
instance sees it).

Relationship to the existing rootfs flow

--image and --rootfs-dir keep working unchanged. Internally they reuse the
same primitives (build, allocate, host-mount, iomem-discover) as kerf shm create, keyed by instance name as today, so there is one code path for
build, mount, and teardown. Rootfs images are not listed by kerf shm ls and
are not attachable.

Code changes

• src/kerf/daxfs/mkdaxfs.py: split create_daxfs_image into primitives
  parameterized by name and mount directory: build image (existing
  DaxfsBuilder), obtain backing, write, host-mount, discover. Add an
  empty-image mode (superblock, empty root inode, overlay region sized from
  --size), and a phys=/size= host-mount variant alongside the dmabuf-fd
  one.
• New backing abstraction (small: obtain a writable mapping plus phys/size,
  release) with dmabuf and cma implementations; src/kerf/lazy_cma.py grows
  mmap and free wrappers for the new module ioctls, failing with a clear
  message on kernels that lack them.
• New src/kerf/shm/ command package (create, ls, show, rm,
  attach, detach) following the existing per-command layout, plus shm
  metadata helpers mirroring metadata.py's atomic-write pattern.
• New agent-protocol module: SET_SHM request/response encoding and a
  host-side vsock client (AF_VSOCK plus the SO_VM_SOCKETS_TRANSPORT
  setsockopt), used by exec, attach, and detach.
• src/kerf/load/main.py: --shm name resolution and validation, attachment
  recording in metadata.
• src/kerf/exec/main.py: after triggering boot, connect to the instance
  agent with retry, push the desired set, report mount status.
• src/kerf/unload/main.py, src/kerf/delete/main.py: detach bookkeeping.
• src/kerf/show/main.py: display attached shm per instance.
• src/kerf/cli.py: register the shm group.
• kerf-init (Makefile-built binary): vsock listener on KERF_AGENT_PORT,
  SET_SHM reconciliation of /shm/, fork the entrypoint after the initial
  push, stay resident reaping children and serving requests.

Testing

• DaxfsBuilder empty-image mode: superblock and overlay header fields.
• kerf shm create validation: name rules (incl. the lazy_cma name-length
  bound), duplicate names, size vs --from-dir exclusivity, --node
  rejected without --backing=cma.
• Backing selection and rm behavior per backing, with mocked device ioctls,
  including the clear-error path when lazy_cma lacks mmap/free.
• --shm resolution: unknown names, duplicates, stale regions.
• Agent protocol encoding: SET_SHM request/response round-trips,
  per-name outcome parsing, protocol version mismatch.
• Host client behavior against a mocked AF_VSOCK socket: connect retry on
  exec, bounded wait, error reporting.
• Attach/detach metadata transitions across load/attach/detach/unload/delete,
  including attach to a loaded-but-not-running instance (metadata only,
  pushed at next exec).
• Stale-region detection with a faked /proc/iomem reading.
• Mount-path behavior needs root and follows the existing daxfs test approach.

Kernel-side dependencies

• Multi-writer daxfs (cross-kernel overlay coordination, format bump) for
  general shared use of an shm by multiple writers. (The mmap-a-file
  plain-memory sharing pattern additionally relies on in-place DAX mmap
  writes to allocated blocks.)
• Multikernel vsock transport (already in the tree: linux commit
  7b3d04bf133e53f2), enabled in both host and spawn kernel configs.
• lazy_cma module: mmap support and a free ioctl, required by the cma
  backing. kerf degrades gracefully (clear errors) where these are missing.

Out of scope

• The raw kind: still deferred, but no longer blocked on a pin mechanism,
  since cma backing is pinned by the kernel resource tree; raw becomes
  mostly CLI and cmdline plumbing once needed.
• The extern backing (adopting arbitrary pre-existing phys+size regions).
• Shared read-only base rootfs across instances.
• Device-tree registration of shm regions for validation against instance
  memory (compatible follow-up).
• Resizing an existing shm.