feat(hotblocks): expose the LSM base level size - #107
Merged
Conversation
Raising write_buffer_size 8x without touching this left L0 arriving larger than the level it merges into: four 512 MB memtables flush ~512 MB of compressed L0 into a 256 MB L1, so every L0 compaction rewrites all of L1 and immediately pushes it down. The canonical pairing is base >= level0_file_num_compaction_trigger x the compressed flush size. It is also the lever the memtable change could not reach. A byte is rewritten once per level, and memtable size only addresses the L0->L1 hop; at ~60 GB live the ladder runs several levels deeper and that work is untouched. The production numbers show it: the bench predicted 2.4x fewer device writes at ~130 MB live, internal delivered 1.67x at ~60 GB. Default stays RocksDB's 256 MB, so this is inert until a deployment sets it. Deliberately unmeasured: the bench cannot price it at 130 MB live, where there is barely a ladder to shorten. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds
--rocksdb-level-base-mb. Default stays RocksDB's own 256 MB, so this is inert until a deployment sets it — same pattern as the memtable flags in #106.Why now
Two independent reasons, both surfaced by the production roll of #106.
1. The memtable change left the ladder mis-sized. Raising
write_buffer_size64 → 512 MB without touching the base inverted the L0/L1 relationship:max_bytes_for_level_base)L0 now arrives larger than the level it merges into, so every L0 compaction rewrites all of L1 and immediately pushes it down. Canonical pairing is base ≥
level0_file_num_compaction_trigger× compressed flush size.2. It is the lever the memtable could not reach. A byte is rewritten once per level it descends. Memtable size only addresses the L0→L1 hop; at ~60 GB live the ladder runs several levels deeper and that work is untouched by it.
Production bears this out. The bench predicted 2.4× fewer device writes — measured at ~130 MB live, where there is barely a ladder. What internal actually delivered at ~60 GB live, 35 min post-roll:
morpho, which writes least and is therefore WAL-dominated, got only −23 % — consistent with the same model from the other end.
The write-stall fix from #106 is unambiguous and unaffected by any of this:
write_stopped1.206 % → 0 on internal-db-1,immutable_memtables1 of 4 across all eight pods, zero restarts on the roll.What this PR deliberately does not do
It does not set a value, and the value is unmeasured. The bench cannot price this parameter: at 130 MB live there is no ladder to shorten, so a sweep there would measure noise. Pricing it needs a bench run with live size in the GBs — materially longer than the runs behind #106 — or a canary on internal.
Suggested starting point when someone does measure it: 2 GB and 4 GB against the current 256 MB, all at
512 × 4+ direct I/O.Cost of raising it
Fewer levels ⇒ lower write amplification, but larger individual compactions, more transient space during them, and read amplification roughly flat or slightly better. Nothing here is free at 60 GB live, which is exactly why the default is unchanged.
🤖 Generated with Claude Code