feat(hotblocks): expose CF_TABLES memtable size and count

crates/hotblocks/src/cli.rs

@@ -78,6 +78,19 @@ pub struct CLI {
     #[arg(long, value_name = "N")]
     pub rocksdb_max_background_jobs: Option<usize>,
 
+    /// `CF_TABLES` memtable size. RocksDB's 64 MB default flushes small L0 files
+    /// continuously; the resulting compaction churn is most of the device write
+    /// bill, and writes stop whenever the flush of one buffer has not finished
+    /// before the next fills (measured: `immutable_memtables` pegged at the
+    /// ceiling on every stalled pod). See docs/measurements/2026-07-27-*.
+    #[arg(long, value_name = "MB", default_value = "64")]
+    pub rocksdb_write_buffer_mb: usize,
+
+    /// `CF_TABLES` memtables before writes stop. The default 2 leaves exactly one
+    /// buffer of headroom while its predecessor flushes.
+    #[arg(long, value_name = "N", default_value = "2")]
+    pub rocksdb_max_write_buffers: i32,
+
     /// Rewrite every table SST older than this, collecting dead data that never made a file
     /// tombstone-dense enough for the deletion collector. Lower means faster reclaim and
     /// proportionally more write amplification. 0 disables it, leaving RocksDB's 30-day
@@ -150,6 +163,8 @@ impl CLI {
             .with_max_log_file_size(self.rocksdb_max_log_file_size)
             .with_keep_log_file_num(self.rocksdb_keep_log_file_num)
             .with_periodic_compaction_secs(self.rocksdb_periodic_compaction_secs)
+            .with_write_buffer_mb(self.rocksdb_write_buffer_mb)
+            .with_max_write_buffers(self.rocksdb_max_write_buffers)
             .with_block_hash_index(self.block_hash_index)
             .with_transaction_hash_index(self.transaction_hash_index);
 

crates/storage/src/db/db.rs

@@ -58,6 +58,8 @@ pub struct DatabaseSettings {
     chunk_cache_size: usize,
     data_cache_size: usize,
     with_rocksdb_stats: bool,
+    write_buffer_mb: usize,
+    max_write_buffers: i32,
     direct_io: bool,
     cache_index_and_filter_blocks: bool,
     max_log_file_size: usize,
@@ -84,6 +86,9 @@ impl Default for DatabaseSettings {
             chunk_cache_size: 64,
             data_cache_size: 256,
             with_rocksdb_stats: false,
+            // RocksDB's own defaults; the CLI carries the deployed values
+            write_buffer_mb: 64,
+            max_write_buffers: 2,
             direct_io: false,
             cache_index_and_filter_blocks: false,
             max_log_file_size: 10,
@@ -118,6 +123,16 @@ impl DatabaseSettings {
         self
     }
 
+    pub fn with_write_buffer_mb(mut self, mb: usize) -> Self {
+        self.write_buffer_mb = mb;
+        self
+    }
+
+    pub fn with_max_write_buffers(mut self, n: i32) -> Self {
+        self.max_write_buffers = n;
+        self
+    }
+
     pub fn with_cache_index_and_filter_blocks(mut self, yes: bool) -> Self {
         self.cache_index_and_filter_blocks = yes;
         self
@@ -238,6 +253,10 @@ impl DatabaseSettings {
         }
         // Bound space amplification (default since RocksDB 8.4; pinned -- load-bearing here).
         options.set_level_compaction_dynamic_level_bytes(true);
+        // Scoped to this CF deliberately: it takes essentially all the write volume, so the
+        // memory cost (write_buffer_mb x max_write_buffers) is paid once rather than per CF.
+        options.set_write_buffer_size(self.write_buffer_mb << 20);
+        options.set_max_write_buffer_number(self.max_write_buffers);
         if !self.auto_compactions {
             options.set_disable_auto_compactions(true);
         }

docs/measurements/2026-07-27-rocksdb-memtable-tuning.md

@@ -0,0 +1,105 @@
+# 2026-07-27 — RocksDB memtable tuning: device writes and write stalls
+
+Production runs `CF_TABLES` on RocksDB's stock memtable defaults — `db.rs` sets no
+memtable, L0, file-size or compaction-style option, so `write_buffer_size` is 64 MB and
+`max_write_buffer_number` is 2. Both of the costs that motivated a storage-engine
+replacement turn out to be attributable to that.
+
+## The production stall is the memtable ceiling, not compaction debt
+
+Measured over 7 d on all four stacks (VictoriaMetrics, `hotblocks_rocksdb_*`):
+
+| signal | stalled pods | healthy pods |
+|---|---|---|
+| `write_stopped` max | 1 (internal ×2, testnet-dev ×2) | 0 (mainnet ×2, morpho ×2) |
+| `immutable_memtables` max | **2** | **1** |
+| `files_at_level0` max | 15 | 7–11 |
+| `pending_compaction_bytes` max | 16.2 GB | 2.3–6.0 GB |
+
+`max_write_buffer_number` is 2, so `immutable_memtables == 2` *is* the stop condition, and
+it partitions the pods exactly. The alternatives do not: `files_at_level0` peaks at 15
+against a default `level0_stop_writes_trigger` of 36 and is the same order on both groups,
+and `pending_compaction_bytes` peaks at 5.9 % of the 256 GB default hard limit. Cost today:
+`mainnet-internal-db-1` spends **1.2 % of wall time** write-stopped (~2 h/week),
+`internal-db-0` 0.089 %.
+
+## Bench sweep
+
+`crates/mdbx-spike --engine rocks`, free-run 45 datasets × 2000 chunks = 15.48 GB raw
+churned, on a bare-metal Xeon E-2136 / 2× NVMe md RAID0 box, in docker `--cpus 8
+--memory 16g --cgroupns private`. Device writes are cgroup `io.stat` excluding the stacked
+md/dm majors. All arms carry production's Zstd WAL compression, `max_background_jobs=8`,
+and **direct I/O** — production parity on all three (verified 2026-07-27 that no stack
+passes `--rocksdb-disable-direct-io`, and that `cli.rs` inverts the flag, so prod runs
+`set_use_direct_reads` + `set_use_direct_io_for_flush_and_compaction`).
+
+| run | write_buffer | max_buffers | device writes | amp vs raw | write_stopped | max immutable | commit p99/max |
+|---|---|---|---|---|---|---|---|
+| 27 | 64 MB | 2 | 47.73 GB | 3.08× | 14.26 % | 2 (ceiling 2) | 160 / 2779 ms |
+| 28 | 64 MB | 2 | 39.07 GB | 2.52× | 5.97 % | 2 (ceiling 2) | 83 / 1569 ms |
+| 35 | 128 MB | 4 | 38.26 GB | 2.47× | **0.00 %** | 2 (ceiling 4) | — |
+| 36 | 256 MB | 4 | 23.45 GB | 1.51× | 0.00 % | 2 (ceiling 4) | — |
+| 37 | 512 MB | 2 | 16.29 GB | 1.05× | 0.00 % | 1 (ceiling 2) | — |
+| 29 | 512 MB | 4 | **16.28 GB** | 1.05× | 0.00 % | 1 (ceiling 4) | 56 / 118 ms |
+| 39 | 1024 MB | 4 | 11.98 GB | 0.77× | 0.00 % | 1 (ceiling 4) | — |
+
+Run 27 is buffered I/O; every other arm is direct. Run 28 is therefore the honest
+stand-in for production as deployed today.
+
+Two effects separate cleanly:
+
+- **Buffer count fixes the stall; it does nothing for write volume.** 128 MB × 4 already
+  takes `write_stopped` from 5.97 % to zero while device writes move 39.07 → 38.26 GB
+  (noise). The stall is purely a headroom problem: with a ceiling of 2 there is exactly one
+  spare buffer while its predecessor flushes.
+- **Buffer size drives write volume; count is irrelevant to it.** 512 MB × 2 and 512 MB × 4
+  land at 16.29 vs 16.28 GB. A 64 MB buffer flushes small L0 files continuously and the
+  resulting compaction generations are most of the device bill; an 8× larger buffer flushes
+  8× less often into 8× larger files.
+
+The curve has **not** plateaued at 1024 MB, and 512 MB is not a knee — it is a point on a
+still-falling slope chosen for its memory cost.
+
+## Consequence for ADR 0002
+
+The mdbx arm of the same bench, same build, same session: **13.27 GB** free-run (run 33),
+3.28 GB at production pace (run 34). Against the tuned reference the engine gap is:
+
+| shape | mdbx | rocks 512 MB × 4 | ratio |
+|---|---|---|---|
+| free-run 45 × 2000 | 13.27 GB | 16.28 GB | 1.23× |
+| paced 45 × 600 @ 750 ms | 3.28 GB | 4.69 GB (run 32) | 1.43× |
+
+ADR 0002 gates the port on "device writes dropping ≥ 3×". Against a reference at
+production I/O parity with one memtable change, the measured advantage is 1.2–1.4×, and at
+1024 MB × 4 RocksDB writes **less** than mdbx (11.98 vs 13.27 GB). The gate does not pass.
+
+What tuning does not address, and what an engine decision should now be argued on: commit
+tails (mdbx paced p99 379 µs vs 3743 µs, ~10×) and per-dataset isolation (INV-35 / LIV-8,
+known-violated as GAP-1).
+
+## Recommended values, and what is unmeasured
+
+Ship `--rocksdb-write-buffer-mb 512 --rocksdb-max-write-buffers 4`. It removes the stall
+class outright and cuts device writes ~2.4× against production as deployed, at 2 GB of
+memtable — scoped to `CF_TABLES`, which takes essentially all the write volume, so the cost
+is paid once rather than per column family.
+
+Not measured, and required before or during rollout:
+
+- **Boot time.** A larger memtable means a longer WAL replay at startup. This service has a
+  documented startup-probe kill-loop at a 90 s budget, so the recovery path must be timed
+  on internal before mainnet.
+- **Level sizing.** `max_bytes_for_level_base` is still at the 256 MB default, now smaller
+  than a single memtable. The textbook pairing is L1 ≈ `level0_file_num_compaction_trigger`
+  × `write_buffer_size`. The numbers above were obtained *without* touching it, so they are
+  a floor on what tuning can do, not a ceiling.
+- **1024 MB.** Better on writes (11.98 GB) but 4 GB of memtable against a 12 G request, and
+  a proportionally longer replay. Revisit once boot time is known.
+- **Scale.** Bench live is ~130 MB against production's ~62 GB. The stall mechanism is
+  scale-free (it is a flush-vs-fill race), but the write-volume ratios are not guaranteed to
+  hold at production live size.
+
+Production prediction to verify on rollout: device writes per mainnet pod ~397 MB/s today
+(corrected figure — see `2026-07-27-production-sli-baseline.md` for the double-counting
+traps in `container_fs_*`) should fall to roughly 140–170 MB/s.