Motivation
daxfs is the persistence layer for multikernel crash recovery: a training job
points its checkpoint directory at a daxfs mount backed by a preserved memory
region, the primary kernel panics, a spawn kernel takes over, mounts the same
region by phys=/size=, and the job auto-resumes from the newest checkpoint
file. No framework changes are needed because every major stack (raw
torch.save, Lightning, HuggingFace, DeepSpeed, Megatron) commits checkpoints
with the same idiom:
write ckpt.tmp # seconds to minutes, gigabytes
rename ckpt.tmp -> ckpt # the commit point
The correctness invariant this idiom relies on: at every instant, the name
ckpt refers to a complete checkpoint. POSIX guarantees it: rename() over
an existing target atomically replaces it, and ext4/xfs/btrfs/tmpfs all honor
that. Applications are entitled to assume it.
Current behavior
daxfs_rename() implements replace as three separate overlay operations:
daxfs_unlink() the existing target
- create the new dirent
- delete the source dirent
A kernel crash between step 1 and step 2 leaves the name pointing at
nothing: the old checkpoint is tombstoned, the new one is still named
ckpt.tmp, and the resume scan on the recovered kernel finds no checkpoint at
all. The job falls back to a cold restart from remote storage, which is
exactly the failure this whole design exists to prevent, triggered at exactly
the moment it is supposed to handle.
"The window is tiny" is not a defense here: crash recovery is a correctness
feature, checkpoints recur every few minutes for the lifetime of the job, and
checkpoint activity (memory pressure, I/O burst) is not uncorrelated with
crash triggers.
Proposed fix
Every overlay operation already publishes through a single CAS, so individual
ops are crash-atomic; the primitive to repoint an existing dirent at a child
inode already exists in overlay.c (used to clear tombstones). Implement
replace-rename on top of it:
- CAS the existing target dirent's inode pointer from the old inode to the
new inode (single published store: this is the commit)
- tombstone the source dirent
- free the old target inode and its data pages
A crash between steps leaves the file visible under both names (benign,
cleanable at next mount) or leaks the old inode's blocks (benign, a
mount-time scavenge can reclaim). At no instant does the target name dangle.
The non-replace rename path (target does not exist) already commits via the
single dirent-create CAS and needs no change.
Notes
- This also restores the POSIX
rename() contract in general, independent of
the crash-recovery use case; silently weaker-than-POSIX semantics under the
same syscall will surface as unreproducible lost files for any application
using the tmp+rename idiom.
RENAME_EXCHANGE can remain unsupported; only the replace path needs to be
atomic.
- A follow-up (separate issue material): a mount-time crash-consistency
scavenger that reclaims never-linked pool entries and duplicate-name
leftovers from a writer that died mid-operation.
Motivation
daxfs is the persistence layer for multikernel crash recovery: a training job
points its checkpoint directory at a daxfs mount backed by a preserved memory
region, the primary kernel panics, a spawn kernel takes over, mounts the same
region by
phys=/size=, and the job auto-resumes from the newest checkpointfile. No framework changes are needed because every major stack (raw
torch.save, Lightning, HuggingFace, DeepSpeed, Megatron) commits checkpointswith the same idiom:
The correctness invariant this idiom relies on: at every instant, the name
ckptrefers to a complete checkpoint. POSIX guarantees it:rename()overan existing target atomically replaces it, and ext4/xfs/btrfs/tmpfs all honor
that. Applications are entitled to assume it.
Current behavior
daxfs_rename()implements replace as three separate overlay operations:daxfs_unlink()the existing targetA kernel crash between step 1 and step 2 leaves the name pointing at
nothing: the old checkpoint is tombstoned, the new one is still named
ckpt.tmp, and the resume scan on the recovered kernel finds no checkpoint atall. The job falls back to a cold restart from remote storage, which is
exactly the failure this whole design exists to prevent, triggered at exactly
the moment it is supposed to handle.
"The window is tiny" is not a defense here: crash recovery is a correctness
feature, checkpoints recur every few minutes for the lifetime of the job, and
checkpoint activity (memory pressure, I/O burst) is not uncorrelated with
crash triggers.
Proposed fix
Every overlay operation already publishes through a single CAS, so individual
ops are crash-atomic; the primitive to repoint an existing dirent at a child
inode already exists in
overlay.c(used to clear tombstones). Implementreplace-rename on top of it:
new inode (single published store: this is the commit)
A crash between steps leaves the file visible under both names (benign,
cleanable at next mount) or leaks the old inode's blocks (benign, a
mount-time scavenge can reclaim). At no instant does the target name dangle.
The non-replace rename path (target does not exist) already commits via the
single dirent-create CAS and needs no change.
Notes
rename()contract in general, independent ofthe crash-recovery use case; silently weaker-than-POSIX semantics under the
same syscall will surface as unreproducible lost files for any application
using the tmp+rename idiom.
RENAME_EXCHANGEcan remain unsupported; only the replace path needs to beatomic.
scavenger that reclaims never-linked pool entries and duplicate-name
leftovers from a writer that died mid-operation.