Skip to content

Make replace-rename atomic so the tmp+rename checkpoint commit survives a kernel crash #28

Description

@congwang-mk

Motivation

daxfs is the persistence layer for multikernel crash recovery: a training job
points its checkpoint directory at a daxfs mount backed by a preserved memory
region, the primary kernel panics, a spawn kernel takes over, mounts the same
region by phys=/size=, and the job auto-resumes from the newest checkpoint
file. No framework changes are needed because every major stack (raw
torch.save, Lightning, HuggingFace, DeepSpeed, Megatron) commits checkpoints
with the same idiom:

write  ckpt.tmp            # seconds to minutes, gigabytes
rename ckpt.tmp -> ckpt    # the commit point

The correctness invariant this idiom relies on: at every instant, the name
ckpt refers to a complete checkpoint. POSIX guarantees it: rename() over
an existing target atomically replaces it, and ext4/xfs/btrfs/tmpfs all honor
that. Applications are entitled to assume it.

Current behavior

daxfs_rename() implements replace as three separate overlay operations:

  1. daxfs_unlink() the existing target
  2. create the new dirent
  3. delete the source dirent

A kernel crash between step 1 and step 2 leaves the name pointing at
nothing: the old checkpoint is tombstoned, the new one is still named
ckpt.tmp, and the resume scan on the recovered kernel finds no checkpoint at
all. The job falls back to a cold restart from remote storage, which is
exactly the failure this whole design exists to prevent, triggered at exactly
the moment it is supposed to handle.

"The window is tiny" is not a defense here: crash recovery is a correctness
feature, checkpoints recur every few minutes for the lifetime of the job, and
checkpoint activity (memory pressure, I/O burst) is not uncorrelated with
crash triggers.

Proposed fix

Every overlay operation already publishes through a single CAS, so individual
ops are crash-atomic; the primitive to repoint an existing dirent at a child
inode already exists in overlay.c (used to clear tombstones). Implement
replace-rename on top of it:

  1. CAS the existing target dirent's inode pointer from the old inode to the
    new inode (single published store: this is the commit)
  2. tombstone the source dirent
  3. free the old target inode and its data pages

A crash between steps leaves the file visible under both names (benign,
cleanable at next mount) or leaks the old inode's blocks (benign, a
mount-time scavenge can reclaim). At no instant does the target name dangle.

The non-replace rename path (target does not exist) already commits via the
single dirent-create CAS and needs no change.

Notes

  • This also restores the POSIX rename() contract in general, independent of
    the crash-recovery use case; silently weaker-than-POSIX semantics under the
    same syscall will surface as unreproducible lost files for any application
    using the tmp+rename idiom.
  • RENAME_EXCHANGE can remain unsupported; only the replace path needs to be
    atomic.
  • A follow-up (separate issue material): a mount-time crash-consistency
    scavenger that reclaims never-linked pool entries and duplicate-name
    leftovers from a writer that died mid-operation.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions