Support checkpoint and restore on RISCV

[congwang-mk]

Currently checkpoint and restore are only implemented on x86_64. We need to make it work for RISCV and ARM64. This issue only tracks the RISCV porting, which has a higher priority than ARM.

[sachin2605]

@congwang-mk Please review.

Design: RISCV64 Checkpoint/Restore Port (Issue #183)

Motivation

The checkpoint/restore engine is fully functional on x86_64 but gated to that
single architecture. Issue #183 requests a riscv64 port. The capture side
already has riscv64 branches for ptrace_getregs and ptrace_getfpregs — the
remaining gap is (1) the restore-side architecture-specific code in
restore_blob.rs, (2) the C restore stub, (3) the sandbox gate that rejects
non-x86_64, and (4) build.rs cross-compilation.

Architecture overview

The checkpoint system has six files:

crates/sandlock-core/src/checkpoint/
├── mod.rs            # Checkpoint struct, enums, constants
├── capture.rs        # Ptrace capture, /proc parsing, process_vm_readv
├── image.rs          # On-disk save/load serialization
├── restore_blob.rs   # Blob planning, region/fd/vdso tables, arch-specific fns
├── resume.rs         # Supervisor restore driver, handshake, process_vm_writev
└── restore-stub.c    # Freestanding C stub, linked at fixed address

Plus build.rs which compiles the stub with -Wl,-Ttext-segment=0x30000000000.

Capture flow: seize → ptrace_getregs → ptrace_getfpregs → parse /proc/pid/maps → process_vm_readv memory → parse /proc/pid/fd → detach

Restore flow: plan() builds blob → fork child with policy → execve restore-stub → stub mmaps regions, reopens fds, relocates vdso, signals READY → supervisor process_vm_writev anon pages → parse current maps → plan sweep → send GO → stub unmaps leftovers, mprotects code, rt_sigreturn into checkpoint

Architecture-specific code: current state

capture.rs

ptrace_getregs has three #[cfg] branches (x86_64: 27 u64 PTRACE_GETREGS,
aarch64: 34 u64 NT_PRSTATUS, riscv64: 32 u64 NT_PRSTATUS) plus a fallback
that returns Unsupported. The riscv64 branch is complete.

ptrace_getfpregs uses NT_X86_XSTATE (0x202, 64 KiB) on x86_64 with
NT_PRFPREG (2, 512 bytes) fallback. On non-x86, it uses NT_PRFPREG (2,
4096 bytes). The riscv64 path hits the non-x86 branch which is adequate.

restore_blob.rs

build_fpstate_image is #[cfg(target_arch = "x86_64")] — builds a signal-frame
compatible xstate image with FP_XSTATE_MAGIC1/2 framing, compaction check, and
fx_only fallback. The non-x86 stub returns Vec::new(). For riscv64, the raw
NT_PRFPREG bytes need to be forwarded to the stub for placement in sc_fpregs.

rearm_restartable_syscall is #[cfg(target_arch = "x86_64")] — checks RAX
for restart sentinels, rewinds RIP by 2, restores ORIG_RAX. The non-x86 stub
is a no-op. For riscv64, a0 holds the sentinel and a7 holds the original
syscall number; sepc rewinds by 4.

sandbox.rs

restore_interactive (L1080) gates with cfg!(not(target_arch = "x86_64")),
returning "checkpoint restore is only implemented on x86_64". This is a runtime
check that must be relaxed to also allow riscv64.

resume.rs

Two tests are #[cfg(target_arch = "x86_64")]:

• stub_links_at_the_reserved_base: checks that the compiled stub binary's
  PT_LOAD segments fall within STUB_BASE..STUB_BASE+STUB_SPAN. The riscv64
  stub must meet the same constraint.
• restore_stub_reconstructs_a_synthetic_image: builds a synthetic one-page
  "program" in x86_64 machine code, checkpoints it, restores through the stub,
  and reads a sentinel byte. This needs a riscv64 code page generator.

write_child_mem and process_vm_writev have a documented assumption about
address space limits on riscv64: the test comment notes that "high addresses
that are free on x86_64 do not exist on riscv64, whose Sv39 user space ends at
256 GiB." The stub's fixed base at 0x300_0000_0000 (192 GiB) is within Sv39's
256 GiB user space, so this is fine.

restore-stub.c

The full x86_64 C stub needs a riscv64 counterpart for:

• Syscall numbers (different between architectures)
• Syscall calling convention (a7 + ecall vs rax + syscall)
• Register file layout (32 u64 vs 27 u64)
• Signal frame layout (different struct rt_sigframe)
• Entry point assembly (mv + la + call vs xor + mov + call)
• dup3 instead of dup2 (riscv64 has no SYS_dup2)
• FP state handling (raw copy vs xrstor)
• No fs_base/set_thread_area needed (tp is in the signal frame)

Design decisions

Decision 1: Separate C stub vs #ifdef

Chosen: Single restore-stub.c with #ifdef __riscv / #elif __x86_64__
preprocessor guards.

Rationale: The stub shares ~70% of its logic between architectures (blob
parsing, READY/GO protocol, mmap/munmap loops, fd reopen, vdso relocation,
sweep logic, string table traversal). Splitting into separate files would
duplicate all shared code and create a maintenance burden for bugfixes.

Decision 2: FP state: pass-through vs framing

Chosen: Pass raw NT_PRFPREG bytes through unchanged, with the stub copying
them directly into the signal frame's sc_fpregs.

Rationale: Unlike x86_64 where the xstate needs SW_MAGIC framing to tell
the kernel to use xrstor vs fxrstor, riscv64's kernel checks the sstatus.FS
field (part of sc_regs) to decide whether to restore FP registers. The ptrace
capture format (NT_PRFPREG) is byte-for-byte identical to the signal frame's
sc_fpregs union — both are struct __riscv_d_ext_state. No magic words,
no format conversion, no size negotiation. If the FPU is disabled (FS=0),
the kernel ignores sc_fpregs entirely.

Rejected alternative: Framing with a size prefix. This would let the stub
validate the size, but the stub already has fpstate_len in the blob header.
An extra framing layer adds complexity with no benefit.

Decision 3: Restartable syscall rearming

Chosen: Implement rearm_restartable_syscall for riscv64 with the same
sentinel values (-512, -513, -514, -516) and a 4-byte rewind.

Rationale: The restart sentinels are kernel-defined (linux/include/
linux/errno.h internal), not architecture-specific. The ecall instruction
on riscv64 is always 4 bytes (compressed instructions cannot encode ecall,
per the RISC-V spec vol 1 12.3), so a 4-byte rewind of sepc is always correct.

Rejected alternative: Leave it as a no-op. While this is simpler, it
means interrupted syscalls (especially nanosleep/ppoll in interactive
workloads) would resume with a stale error code in a0 and an off-by-one
instruction pointer, causing crashes or incorrect behavior. The fix is small
and well-understood.

Decision 4: stub_links_at_the_reserved_base test

Chosen: Generalize the existing x86_64 test to riscv64 by changing the
cfg gate, without changing the test logic.

Rationale: The test parses ELF64 program headers, which is
architecture-agnostic. The STUB_BASE and STUB_SPAN constants are the same for
both architectures (0x300_0000_0000 + 4 MiB). The test only needs to verify
the compiled stub lands within the reserved window.

Decision 5: Synthetic-image test

Chosen: Add a separate #[cfg(target_arch = "riscv64")] version of
restore_stub_reconstructs_a_synthetic_image with riscv64 machine code.

Rationale: The test is heavily architecture-specific: it hand-assembles
a write+exit payload, fills a register file with architecture-specific
indices, and expects the stub to restore to the correct RIP. The blob format
and handshake protocol are the same, but the code page, register layout,
and register file size differ.

Signal frame layout (riscv64)

The kernel's struct rt_sigframe (arch/riscv/kernel/signal.c):

sp →
┌──────────────────────┐
│  struct siginfo      │  (128 bytes)
├──────────────────────┤
│  struct ucontext     │
│  ┌────────────────┐  │
│  │ uc_flags        │  │  (8 bytes)
│  │ uc_link         │  │  (8 bytes)
│  │ uc_stack        │  │  (24 bytes: ss_sp, ss_flags, ss_size)
│  │ uc_sigmask      │  │  (8 bytes)
│  │ uc_mcontext     │  │
│  │ ┌────────────┐  │  │
│  │ │ sc_regs    │  │  │  (32 × 8 = 256 bytes)
│  │ │ sc_fpregs  │  │  │  (union __riscv_fp_state, up to 264 bytes)
│  │ └────────────┘  │  │
│  └────────────────┘  │
└──────────────────────┘

The stub builds this frame on the checkpoint's stack pointer and calls
rt_sigreturn (syscall 139), which never returns.

Blob format impact

No changes needed. The blob format (BLOB_MAGIC="SLRB", BLOB_VERSION=2,
little-endian, HEADER_LEN=64) is architecture-agnostic:

• Registers: Vec<u64> — u64 width matches all 64-bit ISAs; the count is
  regs_len / 8 which the stub reads from the header
• Region entries: start, end, prot, src discriminant, file offset, path offset
• FD entries: fd number, flags, offset, path offset
• VDSO entries: delta (signed i64 from vDSO base), len, target base
• FP state: raw bytes with length in header

The riscv64 stub reads the same header layout and interprets it identically.
Register count difference (27 vs 32) is handled by the header's regs_len field
which the stub uses to calculate the fpstate offset.

Implementation order

1. restore_blob.rs: Add riscv64 rearm_restartable_syscall and
   build_fpstate_image (Steps 1-2). These are small, self-contained, and
   compile-testable on x86_64 via --target riscv64gc-unknown-linux-gnu.

2. sandbox.rs: Relax the restore gate (Step 3). One-line change.

3. restore-stub.c: Add riscv64 #ifdef sections (Step 4). This is the
   largest piece and the most likely to need iteration.

4. build.rs: Add riscv64 stub compilation (Step 5). Required before
   tests can run.

5. resume.rs tests: Update cfg gates and add riscv64 code page (Step 6).

6. Integration tests: Extend if CI has riscv64 (Step 7).

7. Manual verification: On real riscv64 hardware (Step 8).

Open questions

1. FP register size detection: Should the stub detect whether the
   checkpoint's fpstate is the D-extension (264 bytes) or F-extension
   (132 bytes) format, or is it sufficient to trust fpstate_len and memcpy?

   → Trust fpstate_len. The kernel's NT_PRFPREG always returns the
   hardware's native format, and a same-ISA restore means the stub's kernel
   expects the same format.

2. Cross-compilation in CI: Does the CI have riscv64 cross-compilers?
   If not, the build.rs change may need a fallback that skips stub
   compilation on unsupported targets (similar to how a missing C compiler
   is currently handled).

3. VDSO relocation on Sv39: The vDSO base is typically in the high
   portion of the 256 GiB user address space. If the checkpoint's recorded
   vDSO base overlaps the stub's reserved window (0x300_0000_0000), the
   mremap would fail. Is this a realistic collision?

   → Unlikely. On Sv39, the vDSO is mapped near the top of the 256 GiB
   range (around 0x3F_FFFF_0000), while the stub is at 0x300_0000_0000
   (192 GiB). A 64 GiB gap separates them.

4. Signal frame struct layout verification: The struct rt_sigframe
   offsets are hardcoded in the stub. Should there be a compile-time check
   (e.g., static_assert on offsetof) to catch kernel ABI drift?

   → Recommended for the x86_64 stub too. Can be done as a follow-up
   separate from this port, since a kernel ABI break here would equally
   break x86_64.

[sachin2605]

#192

[congwang-mk]

Thanks for the PR, @sachin2605 . Since it is merged, we can close this issue now.