Stop allocating the whole raft log on every WAL append - #130
Merged
Conversation
Two allocations on the persist path still scaled with the length of the raft log, long after the bytes written stopped doing so. Both showed up as malloc dominating a profile of a leader under write load. A deployed process persists only its own node, but the persist loop still built a record for every node first -- each one cloning that node's whole log -- and then skipped all but one. On a three-node group that is three full-log clones per persist with two thrown away. It now builds only the record it is going to write. Encoding an incremental record cloned the entire record and then retained the handful of entries that were actually new. The entries are index-ordered, so the new ones are a binary search and a slice away. Measured on three c7i.large, 3 shards x 3 replicas, with the store held at a fixed 100 keys so only the log grows: growth in write p50 per log entry 0.0030 ms -> 0.0020 ms fdatasync per sync write 2.4 -> 2.1 write p50 (single writer) unchanged at ~26 ms Against the same measurement before any of the WAL work, the growth per entry was 0.0535 ms. What remains is roughly a quarter of it, and the rest of the growth in an ordinary workload is the store getting larger rather than the log.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Two allocations on the persist path still scaled with the length of the raft log, long after
the bytes written stopped doing so. Both showed up as
mallocdominating a profile of aleader under write load.
for every node first — each one cloning that node's whole log — and then skipped all
but one. On a three-node group that is three full-log clones per persist, two of them
thrown away. It now builds only the record it is going to write.
entries that were actually new. Entries are index-ordered, so the new ones are a binary
search and a slice away.
Measured
Three c7i.large, 3 shards × 3 replicas, 30-byte values. The store is held at a fixed 100
keys so that only the log grows — otherwise the store getting larger is measured too.
For scale: the same measurement before any of the incremental-record work grew at 0.0535 ms
per entry, so what is left is under a twentieth of it. In an ordinary workload the remaining
growth is mostly the store getting larger rather than the log — a profile points at the
storage engine's per-write bucket bookkeeping, which is a separate thing to chase.
Also verified unchanged on the same cluster: a killed leader is replaced and writes resume in
25 ms, all acknowledged keys stay readable on both survivors, and all nine replicas serve
byte-exact reads.
Raft suite: 193 passed, 1 failed — the failure is
http_raft_transport_sends_append_vote_and_snapshot_over_tcp,which binds a fixed port and passes on its own (2/2); the suite's fixed ports make it flaky
under a full run.