Skip to content

Stop allocating the whole raft log on every WAL append - #130

Merged
matrixarkai merged 1 commit into
mainfrom
oss/raft-wal-append-allocation
Aug 22, 2026
Merged

Stop allocating the whole raft log on every WAL append#130
matrixarkai merged 1 commit into
mainfrom
oss/raft-wal-append-allocation

Conversation

@bjmeetsfo

Copy link
Copy Markdown
Collaborator

Two allocations on the persist path still scaled with the length of the raft log, long after
the bytes written stopped doing so. Both showed up as malloc dominating a profile of a
leader under write load.

  • A deployed process persists only its own node, but the persist loop still built a record
    for every node first — each one cloning that node's whole log — and then skipped all
    but one. On a three-node group that is three full-log clones per persist, two of them
    thrown away. It now builds only the record it is going to write.
  • Encoding an incremental record cloned the entire record and then retained the handful of
    entries that were actually new. Entries are index-ordered, so the new ones are a binary
    search and a slice away.

Measured

Three c7i.large, 3 shards × 3 replicas, 30-byte values. The store is held at a fixed 100
keys so that only the log grows — otherwise the store getting larger is measured too.

before this change after
growth in write p50, per log entry 0.0030 ms 0.0020 ms
fdatasync per sync write 2.4 2.1
write p50, single writer 26.2 ms 26.0 ms

For scale: the same measurement before any of the incremental-record work grew at 0.0535 ms
per entry, so what is left is under a twentieth of it. In an ordinary workload the remaining
growth is mostly the store getting larger rather than the log — a profile points at the
storage engine's per-write bucket bookkeeping, which is a separate thing to chase.

Also verified unchanged on the same cluster: a killed leader is replaced and writes resume in
25 ms, all acknowledged keys stay readable on both survivors, and all nine replicas serve
byte-exact reads.

Raft suite: 193 passed, 1 failed — the failure is http_raft_transport_sends_append_vote_and_snapshot_over_tcp,
which binds a fixed port and passes on its own (2/2); the suite's fixed ports make it flaky
under a full run.

Two allocations on the persist path still scaled with the length of the raft log, long
after the bytes written stopped doing so. Both showed up as malloc dominating a profile
of a leader under write load.

A deployed process persists only its own node, but the persist loop still built a record
for every node first -- each one cloning that node's whole log -- and then skipped all but
one. On a three-node group that is three full-log clones per persist with two thrown away.
It now builds only the record it is going to write.

Encoding an incremental record cloned the entire record and then retained the handful of
entries that were actually new. The entries are index-ordered, so the new ones are a binary
search and a slice away.

Measured on three c7i.large, 3 shards x 3 replicas, with the store held at a fixed 100 keys
so only the log grows:

  growth in write p50 per log entry   0.0030 ms -> 0.0020 ms
  fdatasync per sync write            2.4 -> 2.1
  write p50 (single writer)           unchanged at ~26 ms

Against the same measurement before any of the WAL work, the growth per entry was 0.0535 ms.
What remains is roughly a quarter of it, and the rest of the growth in an ordinary workload
is the store getting larger rather than the log.
@bjmeetsfo
bjmeetsfo requested a review from superhaiou as a code owner August 22, 2026 08:51
@matrixarkai
matrixarkai merged commit 42feb65 into main Aug 22, 2026
6 checks passed
@matrixarkai
matrixarkai deleted the oss/raft-wal-append-allocation branch August 22, 2026 19:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants